2024-03-21T12:17:53.288 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T12:17:53.288 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_gabrioux@teuthology --name gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi --archive /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915 --verbose --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} -- /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/orig.config.yaml 2024-03-21T12:17:53.331 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T12:17:53.423 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915 branch: wip-guits-testing-2024-03-21-0818 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: gabrioux@ibm.com first_in_suite: false job_id: '7614915' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-2024-03-21-0818 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 960fbbe4645c2becc8084669be194c95183c7753 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 960fbbe4645c2becc8084669be194c95183c7753 workunit: branch: wip-guits-testing-2024-03-21-0818 sha1: 960fbbe4645c2becc8084669be194c95183c7753 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 187 sha1: 960fbbe4645c2becc8084669be194c95183c7753 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-2024-03-21-0818 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 960fbbe4645c2becc8084669be194c95183c7753 targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFF0nGXA1g7Xypq//zN2Ry18gLmBxWsgtx1/hZ86a45K7oECes8HVAz9MMGYVaelcKEbyfowsVmPy6gMIMmhtb0= smithi177.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL9zPSzfwOtgaPVEAQ34yKSvEMWmY3lOplPS0T7dkoNHJJxTZy40JHrGkyrSOUZjsbW8uwiAO8UUurct6X2Ewgs= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-21_12:04:17 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T12:17:53.423 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa; will attempt to use it 2024-03-21T12:17:53.423 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_960fbbe4645c2becc8084669be194c95183c7753/qa/tasks 2024-03-21T12:17:53.424 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T12:17:53.425 INFO:teuthology.task.internal:Checking packages... 2024-03-21T12:17:53.448 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '960fbbe4645c2becc8084669be194c95183c7753' 2024-03-21T12:17:53.449 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T12:17:53.449 INFO:teuthology.packaging:ref: None 2024-03-21T12:17:53.449 INFO:teuthology.packaging:tag: None 2024-03-21T12:17:53.449 INFO:teuthology.packaging:branch: wip-guits-testing-2024-03-21-0818 2024-03-21T12:17:53.449 INFO:teuthology.packaging:sha1: 960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:17:53.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-2024-03-21-0818 2024-03-21T12:17:53.583 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2267.g960fbbe4 2024-03-21T12:17:53.584 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T12:17:53.590 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T12:17:53.590 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T12:17:53.597 INFO:teuthology.task.internal:Saving configuration 2024-03-21T12:17:53.614 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T12:17:53.619 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T12:17:53.644 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:12:17.106097', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFF0nGXA1g7Xypq//zN2Ry18gLmBxWsgtx1/hZ86a45K7oECes8HVAz9MMGYVaelcKEbyfowsVmPy6gMIMmhtb0='} 2024-03-21T12:17:53.663 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:12:17.107610', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL9zPSzfwOtgaPVEAQ34yKSvEMWmY3lOplPS0T7dkoNHJJxTZy40JHrGkyrSOUZjsbW8uwiAO8UUurct6X2Ewgs='} 2024-03-21T12:17:53.663 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T12:17:53.670 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-21T12:17:53.671 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-21T12:17:53.671 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T12:17:53.738 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9e3a79e0d0>, signals=[15]) 2024-03-21T12:17:53.738 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T12:17:53.744 INFO:teuthology.task.internal:Opening connections... 2024-03-21T12:17:53.744 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-21T12:17:53.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:17:53.837 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi177.front.sepia.ceph.com 2024-03-21T12:17:53.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:17:53.925 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T12:17:53.932 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-21T12:17:53.949 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-21T12:17:53.949 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-21T12:17:54.004 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-03-21T12:17:54.004 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-03-21T12:17:54.004 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T12:17:54.005 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T12:17:54.006 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-21T12:17:54.124 DEBUG:teuthology.orchestra.run.smithi177:> uname -m 2024-03-21T12:17:54.142 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2024-03-21T12:17:54.142 DEBUG:teuthology.orchestra.run.smithi177:> cat /etc/os-release 2024-03-21T12:17:54.198 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2024-03-21T12:17:54.198 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="9" 2024-03-21T12:17:54.198 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2024-03-21T12:17:54.198 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="9" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el9" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:LOGO="fedora-logo-icon" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T12:17:54.199 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T12:17:54.199 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2024-03-21T12:17:54.271 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T12:17:54.280 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T12:17:54.285 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T12:17:54.285 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-21T12:17:54.288 DEBUG:teuthology.orchestra.run.smithi177:> test '!' -e /home/ubuntu/cephtest 2024-03-21T12:17:54.304 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T12:17:54.310 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T12:17:54.310 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-21T12:17:54.344 DEBUG:teuthology.orchestra.run.smithi177:> test -z $(ls -A /var/lib/ceph) 2024-03-21T12:17:54.373 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T12:17:54.424 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T12:17:54.438 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-21T12:17:54.439 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-21T12:17:54.439 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T12:17:54.439 DEBUG:teuthology.orchestra.run.smithi177:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T12:17:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:17:54.455 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-21T12:17:54.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:17:54.456 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-21T12:17:54.512 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-361.el9.x86_64 2024-03-21T12:17:54.512 INFO:teuthology.task.kernel:Running kernel on smithi177: 5.14.0-361.el9.x86_64 2024-03-21T12:17:54.512 DEBUG:teuthology.orchestra.run.smithi177:> sudo yum install -y kernel 2024-03-21T12:17:54.513 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-361.el9.x86_64 2024-03-21T12:17:54.513 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-361.el9.x86_64 2024-03-21T12:17:54.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-03-21T12:17:57.095 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 6.9 MB/s | 6.7 MB 00:00 2024-03-21T12:17:57.176 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - AppStream 6.8 MB/s | 6.7 MB 00:00 2024-03-21T12:17:58.787 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-21T12:17:58.975 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-03-21T12:18:02.522 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-21T12:18:03.157 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-21T12:18:04.404 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-21T12:18:09.554 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-21T12:18:11.322 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-21T12:18:13.225 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-21T12:18:18.206 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-21T12:18:20.304 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T12:18:20.304 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T12:18:20.395 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:18:20.404 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-21T12:18:20.405 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-21T12:18:20.405 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-03-21T12:18:20.405 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:18:20.405 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 76 M 2024-03-21T12:18:20.405 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 123 M 2024-03-21T12:18:20.405 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-21T12:18:21.098 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 71 kB/s | 44 kB 00:00 2024-03-21T12:18:27.472 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - AppStream 829 kB/s | 19 MB 00:23 2024-03-21T12:18:38.546 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - CRB 821 kB/s | 6.1 MB 00:07 2024-03-21T12:18:40.138 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 776 kB/s | 15 MB 00:19 2024-03-21T12:18:40.181 INFO:teuthology.orchestra.run.smithi177.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-21T12:18:41.832 INFO:teuthology.orchestra.run.smithi177.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-21T12:18:46.731 INFO:teuthology.orchestra.run.smithi177.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-21T12:18:48.771 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T12:18:48.771 INFO:teuthology.orchestra.run.smithi177.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T12:18:48.858 INFO:teuthology.orchestra.run.smithi177.stdout:Dependencies resolved. 2024-03-21T12:18:48.870 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-21T12:18:48.870 INFO:teuthology.orchestra.run.smithi177.stdout: Package Architecture Version Repository Size 2024-03-21T12:18:48.870 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-21T12:18:48.870 INFO:teuthology.orchestra.run.smithi177.stdout:Installing: 2024-03-21T12:18:48.870 INFO:teuthology.orchestra.run.smithi177.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout:Installing dependencies: 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction Summary 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout:================================================================================ 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout:Install 4 Packages 2024-03-21T12:18:48.871 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:18:48.872 INFO:teuthology.orchestra.run.smithi177.stdout:Total download size: 76 M 2024-03-21T12:18:48.872 INFO:teuthology.orchestra.run.smithi177.stdout:Installed size: 123 M 2024-03-21T12:18:48.872 INFO:teuthology.orchestra.run.smithi177.stdout:Downloading Packages: 2024-03-21T12:18:49.008 INFO:teuthology.orchestra.run.smithi177.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 472 kB/s | 44 kB 00:00 2024-03-21T12:18:49.374 INFO:teuthology.orchestra.run.smithi177.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 32 MB/s | 15 MB 00:00 2024-03-21T12:18:49.766 INFO:teuthology.orchestra.run.smithi177.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 36 MB/s | 27 MB 00:00 2024-03-21T12:18:50.075 INFO:teuthology.orchestra.run.smithi177.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 29 MB/s | 33 MB 00:01 2024-03-21T12:18:50.075 INFO:teuthology.orchestra.run.smithi177.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:18:50.076 INFO:teuthology.orchestra.run.smithi177.stdout:Total 63 MB/s | 76 MB 00:01 2024-03-21T12:18:50.734 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction check 2024-03-21T12:18:51.086 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction check succeeded. 2024-03-21T12:18:51.086 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction test 2024-03-21T12:18:52.077 INFO:teuthology.orchestra.run.smithi177.stdout:Transaction test succeeded. 2024-03-21T12:18:52.077 INFO:teuthology.orchestra.run.smithi177.stdout:Running transaction 2024-03-21T12:18:54.571 INFO:teuthology.orchestra.run.smithi177.stdout: Preparing : 1/1 2024-03-21T12:18:55.188 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:18:55.284 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:18:56.969 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 784 kB/s | 27 MB 00:35 2024-03-21T12:18:58.009 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:18:58.145 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:19:02.843 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:19:02.945 INFO:teuthology.orchestra.run.smithi177.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:03.403 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 798 kB/s | 33 MB 00:42 2024-03-21T12:19:03.404 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-21T12:19:03.404 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.8 MB/s | 76 MB 00:42 2024-03-21T12:19:04.064 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-21T12:19:04.429 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-21T12:19:04.429 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-21T12:19:05.460 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-21T12:19:05.460 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-21T12:19:07.640 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:07.931 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-21T12:19:08.551 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:19:08.642 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:19:11.392 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:19:11.532 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:19:16.258 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:19:16.356 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:21.091 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:36.971 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:36.971 INFO:teuthology.orchestra.run.smithi177.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T12:19:36.971 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:19:37.014 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:37.404 INFO:teuthology.orchestra.run.smithi177.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:37.404 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:19:37.404 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:19:37.404 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:19:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:19:37.821 INFO:teuthology.orchestra.run.smithi177.stdout:Installed: 2024-03-21T12:19:37.821 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T12:19:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T12:19:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T12:19:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T12:19:37.822 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:19:37.822 INFO:teuthology.orchestra.run.smithi177.stdout:Complete! 2024-03-21T12:19:38.154 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:19:38.294 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:19:38.295 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:19:38.295 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T12:19:38.295 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T12:19:38.295 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T12:19:38.295 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-21T12:19:38.295 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi177.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T12:19:38.295 DEBUG:teuthology.orchestra.run.smithi177:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:19:38.374 INFO:teuthology.orchestra.run.smithi177.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:19:38.375 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:19:38.375 DEBUG:teuthology.orchestra.run.smithi177:> sudo rpm -qi grub2-tools 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Name : grub2-tools 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Epoch : 1 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Version : 2.06 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Release : 68.el9 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Architecture: x86_64 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Group : Unspecified 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Size : 8264254 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:License : GPLv3+ 2024-03-21T12:19:38.419 INFO:teuthology.orchestra.run.smithi177.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Packager : builder@centos.org 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Vendor : CentOS 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Summary : Support tools for GRUB. 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:Description : 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T12:19:38.420 INFO:teuthology.orchestra.run.smithi177.stdout:hardware devices. 2024-03-21T12:19:38.421 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:19:38.421 INFO:teuthology.orchestra.run.smithi177.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T12:19:38.422 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T12:19:38.422 INFO:teuthology.task.kernel:Updating grub on smithi177 to boot 5.14.0-430.el9.x86_64 2024-03-21T12:19:38.422 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T12:19:39.001 INFO:teuthology.orchestra.run.smithi177.stderr:Generating grub configuration file ... 2024-03-21T12:19:40.538 INFO:teuthology.orchestra.run.smithi177.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T12:19:40.563 INFO:teuthology.orchestra.run.smithi177.stderr:done 2024-03-21T12:19:40.583 DEBUG:teuthology.orchestra.run.smithi177:> mktemp 2024-03-21T12:19:40.639 INFO:teuthology.orchestra.run.smithi177.stdout:/tmp/tmp.clstYUapX8 2024-03-21T12:19:40.639 DEBUG:teuthology.orchestra.run.smithi177:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.clstYUapX8 2024-03-21T12:19:40.714 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0666 /tmp/tmp.clstYUapX8 2024-03-21T12:19:40.851 DEBUG:teuthology.orchestra.remote:smithi177:/tmp/tmp.clstYUapX8 is 6KB 2024-03-21T12:19:40.900 DEBUG:teuthology.orchestra.run.smithi177:> rm -fr /tmp/tmp.clstYUapX8 2024-03-21T12:19:40.914 DEBUG:teuthology.orchestra.run.smithi177:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T12:19:40.978 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T12:19:40.978 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T12:19:40.978 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T12:19:40.978 INFO:teuthology.orchestra.run.smithi177.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T12:19:40.980 DEBUG:teuthology.orchestra.run.smithi177:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T12:19:41.137 DEBUG:teuthology.orchestra.run.smithi177:> sudo shutdown -r now 2024-03-21T12:19:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:50.303 INFO:teuthology.orchestra.run.smithi172.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T12:19:50.303 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:19:50.347 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:50.681 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T12:19:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T12:19:50.682 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T12:19:51.098 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:19:51.099 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-21T12:19:51.445 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:19:51.553 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:19:51.554 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:19:51.554 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T12:19:51.554 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T12:19:51.554 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T12:19:51.554 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-21T12:19:51.554 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T12:19:51.554 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T12:19:51.634 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T12:19:51.634 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T12:19:51.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 68.el9 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8264254 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T12:19:51.668 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:19:51.669 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T12:19:51.670 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T12:19:51.670 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-430.el9.x86_64 2024-03-21T12:19:51.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T12:19:52.316 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-21T12:19:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T12:19:53.665 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-21T12:19:53.675 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-21T12:19:53.690 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.q67WYlw7Ft 2024-03-21T12:19:53.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.q67WYlw7Ft 2024-03-21T12:19:53.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.q67WYlw7Ft 2024-03-21T12:19:53.895 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.q67WYlw7Ft is 6KB 2024-03-21T12:19:53.943 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.q67WYlw7Ft 2024-03-21T12:19:53.957 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T12:19:54.031 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T12:19:54.031 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T12:19:54.031 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T12:19:54.032 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T12:19:54.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T12:19:54.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-21T12:20:11.154 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-21T12:20:11.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-21T12:20:11.155 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:20:24.202 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-21T12:20:24.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:20:24.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:20:29.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-21T12:20:38.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-21T12:20:38.608 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:20:41.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.177 2024-03-21T12:20:53.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-21T12:20:53.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:20:58.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-21T12:21:07.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:21:07.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:21:25.753 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-21T12:21:26.166 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-21T12:21:26.167 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-21T12:21:26.167 DEBUG:teuthology.orchestra.run.smithi177:> uname -r 2024-03-21T12:21:26.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-21T12:21:26.181 INFO:teuthology.orchestra.run.smithi177.stdout:5.14.0-430.el9.x86_64 2024-03-21T12:21:26.181 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T12:21:26.181 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T12:21:26.181 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T12:21:27.183 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-21T12:21:27.183 DEBUG:teuthology.orchestra.run.smithi177:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T12:21:27.242 INFO:teuthology.orchestra.run.smithi177.stdout:ttyS1 2024-03-21T12:21:27.285 DEBUG:teuthology.parallel:result is None 2024-03-21T12:21:38.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:21:38.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:21:41.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-21T12:21:56.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:21:56.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:21:56.615 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-21T12:21:57.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:21:57.013 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-21T12:21:57.013 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-21T12:21:57.028 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-430.el9.x86_64 2024-03-21T12:21:57.028 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T12:21:57.028 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T12:21:57.028 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T12:21:58.029 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-21T12:21:58.029 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T12:21:58.097 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-03-21T12:21:58.126 DEBUG:teuthology.parallel:result is None 2024-03-21T12:21:58.127 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T12:21:58.133 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T12:21:58.134 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T12:21:58.136 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T12:21:58.155 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T12:21:58.161 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T12:21:58.168 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T12:21:58.168 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T12:21:58.193 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T12:21:58.242 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T12:21:58.249 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T12:21:58.249 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T12:21:58.268 DEBUG:teuthology.orchestra.run.smithi177:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T12:21:58.298 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:21:58.311 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:21:58.323 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:21:58.337 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T12:21:58.339 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T12:21:58.346 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T12:21:58.346 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T12:21:58.356 DEBUG:teuthology.orchestra.run.smithi177:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T12:21:58.386 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T12:21:58.395 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T12:21:58.395 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T12:21:58.424 DEBUG:teuthology.orchestra.run.smithi177:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T12:21:58.442 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:21:58.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:21:58.567 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:21:58.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:21:58.691 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:21:58.691 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T12:21:58.758 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:21:58.804 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T12:21:58.853 DEBUG:teuthology.orchestra.run.smithi177:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:21:58.936 DEBUG:teuthology.orchestra.run.smithi177:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T12:21:58.982 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:21:58.982 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T12:21:59.051 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-21T12:21:59.054 DEBUG:teuthology.orchestra.run.smithi177:> sudo service rsyslog restart 2024-03-21T12:21:59.101 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T12:21:59.146 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T12:21:59.644 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T12:21:59.652 INFO:teuthology.task.internal:Starting timer... 2024-03-21T12:21:59.652 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T12:21:59.664 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T12:21:59.672 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-03-21T12:21:59.725 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-03-21T12:21:59.727 DEBUG:teuthology.orchestra.run.smithi177:> sudo service auditd rotate 2024-03-21T12:21:59.785 INFO:teuthology.orchestra.run.smithi177.stdout:Rotating logs: 2024-03-21T12:21:59.787 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T12:21:59.787 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-03-21T12:21:59.840 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-03-21T12:21:59.841 DEBUG:teuthology.orchestra.run.smithi177:> /usr/sbin/getenforce 2024-03-21T12:21:59.872 INFO:teuthology.orchestra.run.smithi177.stdout:Permissive 2024-03-21T12:21:59.872 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi172.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi177.front.sepia.ceph.com': 'permissive'} 2024-03-21T12:21:59.872 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T12:21:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:21:59.903 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T12:21:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T12:21:59.941 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T12:21:59.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-03-21T12:21:59.976 DEBUG:teuthology.orchestra.run.smithi177:> sudo /usr/sbin/setenforce permissive 2024-03-21T12:22:00.024 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T12:22:00.033 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T12:22:00.051 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T12:22:00.052 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com,smithi177.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T12:26:24.302 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com'), Remote(name='ubuntu@smithi177.front.sepia.ceph.com')] 2024-03-21T12:26:24.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:26:24.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:26:24.394 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-21T12:26:24.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-21T12:26:24.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-21T12:26:24.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi177.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T12:26:24.562 DEBUG:teuthology.orchestra.run.smithi177:> true 2024-03-21T12:26:24.595 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi177.front.sepia.ceph.com' 2024-03-21T12:26:24.596 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T12:26:24.605 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T12:26:24.605 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T12:26:24.606 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T12:26:24.608 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T12:26:24.608 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T12:26:24.638 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T12:26:24.653 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T12:26:24.668 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T12:26:24.682 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-03-21T12:26:24.684 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T12:26:24.695 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-03-21T12:26:24.712 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T12:26:24.713 INFO:teuthology.orchestra.run.smithi177.stderr:sudo: ntpd: command not found 2024-03-21T12:26:24.726 INFO:teuthology.orchestra.run.smithi177.stdout:506 Cannot talk to daemon 2024-03-21T12:26:24.728 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T12:26:24.743 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T12:26:24.759 INFO:teuthology.orchestra.run.smithi177.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T12:26:24.777 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-21T12:26:24.780 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T12:26:24.780 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-21T12:26:24.780 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.780 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.780 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.780 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.811 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-21T12:26:24.816 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T12:26:24.816 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-21T12:26:24.816 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.816 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.817 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.817 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T12:26:24.818 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-21T12:26:24.938 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '960fbbe4645c2becc8084669be194c95183c7753'} 2024-03-21T12:26:24.938 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-21T12:26:24.939 INFO:tasks.cephadm:Cluster fsid is 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:26:24.939 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-21T12:26:24.939 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.172', 'mon.c': '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]', 'mon.b': '172.21.15.177'} 2024-03-21T12:26:24.939 INFO:tasks.cephadm:First mon is mon.a on smithi172 2024-03-21T12:26:24.939 INFO:tasks.cephadm:First mgr is y 2024-03-21T12:26:24.940 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-21T12:26:24.940 DEBUG:teuthology.orchestra.run.smithi172:> sudo hostname $(hostname -s) 2024-03-21T12:26:24.968 DEBUG:teuthology.orchestra.run.smithi177:> sudo hostname $(hostname -s) 2024-03-21T12:26:24.996 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-21T12:26:24.996 DEBUG:teuthology.orchestra.run.smithi172:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T12:26:25.235 INFO:teuthology.orchestra.run.smithi172.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 21 12:26 /home/ubuntu/cephtest/cephadm 2024-03-21T12:26:25.235 DEBUG:teuthology.orchestra.run.smithi177:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T12:26:25.393 INFO:teuthology.orchestra.run.smithi177.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 21 12:26 /home/ubuntu/cephtest/cephadm 2024-03-21T12:26:25.393 DEBUG:teuthology.orchestra.run.smithi172:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T12:26:25.409 DEBUG:teuthology.orchestra.run.smithi177:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T12:26:25.434 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-21T12:26:25.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-21T12:26:25.451 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-21T12:26:26.317 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-21T12:26:26.345 INFO:teuthology.orchestra.run.smithi177.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-21T12:27:00.420 INFO:teuthology.orchestra.run.smithi177.stdout:{ 2024-03-21T12:27:00.420 INFO:teuthology.orchestra.run.smithi177.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-21T12:27:00.421 INFO:teuthology.orchestra.run.smithi177.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-21T12:27:00.421 INFO:teuthology.orchestra.run.smithi177.stdout: "repo_digests": [ 2024-03-21T12:27:00.421 INFO:teuthology.orchestra.run.smithi177.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-21T12:27:00.421 INFO:teuthology.orchestra.run.smithi177.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-21T12:27:00.421 INFO:teuthology.orchestra.run.smithi177.stdout: ] 2024-03-21T12:27:00.421 INFO:teuthology.orchestra.run.smithi177.stdout:} 2024-03-21T12:27:01.349 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "repo_digests": [ 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-21T12:27:01.350 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:27:01.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph 2024-03-21T12:27:01.412 DEBUG:teuthology.orchestra.run.smithi177:> sudo mkdir -p /etc/ceph 2024-03-21T12:27:01.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /etc/ceph 2024-03-21T12:27:01.481 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 777 /etc/ceph 2024-03-21T12:27:01.513 INFO:tasks.cephadm:Writing seed config... 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-21T12:27:01.514 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-21T12:27:01.515 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-21T12:27:01.515 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:27:01.515 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-21T12:27:01.536 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3bbfe794-e77e-11ee-95cd-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-21T12:27:01.536 DEBUG:teuthology.orchestra.run.smithi172:mon.a> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service 2024-03-21T12:27:01.578 DEBUG:teuthology.orchestra.run.smithi172:mgr.y> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service 2024-03-21T12:27:01.622 INFO:tasks.cephadm:Bootstrapping... 2024-03-21T12:27:01.622 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.172 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-21T12:27:01.817 INFO:teuthology.orchestra.run.smithi172.stderr:-------------------------------------------------------------------------------- 2024-03-21T12:27:01.817 INFO:teuthology.orchestra.run.smithi172.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '3bbfe794-e77e-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.172', '--skip-admin-label'] 2024-03-21T12:27:01.840 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: 4.9.3-dev 2024-03-21T12:27:01.843 INFO:teuthology.orchestra.run.smithi172.stderr:Verifying podman|docker is present... 2024-03-21T12:27:01.864 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: 4.9.3-dev 2024-03-21T12:27:01.865 INFO:teuthology.orchestra.run.smithi172.stderr:Verifying lvm2 is present... 2024-03-21T12:27:01.865 INFO:teuthology.orchestra.run.smithi172.stderr:Verifying time synchronization is in place... 2024-03-21T12:27:01.872 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T12:27:01.878 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: inactive 2024-03-21T12:27:01.884 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: enabled 2024-03-21T12:27:01.890 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: active 2024-03-21T12:27:01.891 INFO:teuthology.orchestra.run.smithi172.stderr:Unit chronyd.service is enabled and running 2024-03-21T12:27:01.891 INFO:teuthology.orchestra.run.smithi172.stderr:Repeating the final host check... 2024-03-21T12:27:01.911 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: 4.9.3-dev 2024-03-21T12:27:01.913 INFO:teuthology.orchestra.run.smithi172.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-21T12:27:01.913 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl is present 2024-03-21T12:27:01.913 INFO:teuthology.orchestra.run.smithi172.stderr:lvcreate is present 2024-03-21T12:27:01.920 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T12:27:01.925 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: inactive 2024-03-21T12:27:01.932 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: enabled 2024-03-21T12:27:01.938 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: active 2024-03-21T12:27:01.938 INFO:teuthology.orchestra.run.smithi172.stderr:Unit chronyd.service is enabled and running 2024-03-21T12:27:01.939 INFO:teuthology.orchestra.run.smithi172.stderr:Host looks OK 2024-03-21T12:27:01.939 INFO:teuthology.orchestra.run.smithi172.stderr:Cluster fsid: 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:01.939 INFO:teuthology.orchestra.run.smithi172.stderr:Acquiring lock 140525285838176 on /run/cephadm/3bbfe794-e77e-11ee-95cd-87774f69a715.lock 2024-03-21T12:27:01.939 INFO:teuthology.orchestra.run.smithi172.stderr:Lock 140525285838176 acquired on /run/cephadm/3bbfe794-e77e-11ee-95cd-87774f69a715.lock 2024-03-21T12:27:01.939 INFO:teuthology.orchestra.run.smithi172.stderr:Verifying IP 172.21.15.172 port 3300 ... 2024-03-21T12:27:01.939 INFO:teuthology.orchestra.run.smithi172.stderr:Verifying IP 172.21.15.172 port 6789 ... 2024-03-21T12:27:01.940 INFO:teuthology.orchestra.run.smithi172.stderr:Base mon IP is 172.21.15.172, final addrv is [v2:172.21.15.172:3300,v1:172.21.15.172:6789] 2024-03-21T12:27:01.942 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.172 metric 100 2024-03-21T12:27:01.942 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.172 metric 100 2024-03-21T12:27:01.944 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-21T12:27:01.945 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-21T12:27:01.945 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1731sec hoplimit 64 pref medium 2024-03-21T12:27:01.947 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-21T12:27:01.947 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-21T12:27:01.947 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-21T12:27:01.947 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-21T12:27:01.947 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:7f91/64 scope link 2024-03-21T12:27:01.947 INFO:teuthology.orchestra.run.smithi172.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-21T12:27:01.948 INFO:teuthology.orchestra.run.smithi172.stderr:Mon IP `172.21.15.172` is in CIDR network `172.21.0.0/20` 2024-03-21T12:27:01.949 INFO:teuthology.orchestra.run.smithi172.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-21T12:27:01.949 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-21T12:27:02.050 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-21T12:27:02.429 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Getting image source signatures 2024-03-21T12:27:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-21T12:27:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-21T12:27:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-21T12:27:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-21T12:27:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-21T12:27:02.430 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-21T12:27:02.579 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: Writing manifest to image destination 2024-03-21T12:27:02.681 INFO:teuthology.orchestra.run.smithi172.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-21T12:27:03.084 INFO:teuthology.orchestra.run.smithi172.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-21T12:27:03.621 INFO:teuthology.orchestra.run.smithi172.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-21T12:27:03.622 INFO:teuthology.orchestra.run.smithi172.stderr:Extracting ceph user uid/gid from container image... 2024-03-21T12:27:03.996 INFO:teuthology.orchestra.run.smithi172.stderr:stat: 167 167 2024-03-21T12:27:04.502 INFO:teuthology.orchestra.run.smithi172.stderr:Creating initial keys... 2024-03-21T12:27:04.885 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph-authtool: AQCYJ/xlJ9qgMBAA/TpHhXH8JVs4w9JMrlAiEA== 2024-03-21T12:27:05.732 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph-authtool: AQCZJ/xlCMCEJhAA+JxSoGkd92loP74hfgnvCw== 2024-03-21T12:27:06.604 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph-authtool: AQCaJ/xl58tgHxAAg+uF3wcSJdXPnqQBbojZ0w== 2024-03-21T12:27:07.060 INFO:teuthology.orchestra.run.smithi172.stderr:Creating initial monmap... 2024-03-21T12:27:07.911 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T12:27:07.911 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-21T12:27:07.912 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:07.912 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T12:27:13.565 INFO:teuthology.orchestra.run.smithi172.stderr:monmaptool for a [v2:172.21.15.172:3300,v1:172.21.15.172:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T12:27:13.565 INFO:teuthology.orchestra.run.smithi172.stderr:setting min_mon_release = octopus 2024-03-21T12:27:13.565 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/monmaptool: set fsid to 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:13.565 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T12:27:13.565 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:27:13.565 INFO:teuthology.orchestra.run.smithi172.stderr:Creating mon... 2024-03-21T12:27:18.676 INFO:teuthology.orchestra.run.smithi172.stderr:create mon.a on 2024-03-21T12:27:18.896 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-21T12:27:19.050 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715.target. 2024-03-21T12:27:19.050 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715.target. 2024-03-21T12:27:19.347 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Failed to reset failed state of unit ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-21T12:27:19.355 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715.target.wants/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@.service. 2024-03-21T12:27:20.172 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: disabled 2024-03-21T12:27:20.178 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: inactive 2024-03-21T12:27:20.179 INFO:teuthology.orchestra.run.smithi172.stderr:firewalld.service is not enabled 2024-03-21T12:27:20.179 INFO:teuthology.orchestra.run.smithi172.stderr:Not possible to enable service . firewalld.service is not available 2024-03-21T12:27:20.180 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for mon to start... 2024-03-21T12:27:20.180 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for mon... 2024-03-21T12:27:20.293 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:20 smithi172 systemd[1]: Started Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:27:20.675 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: cluster: 2024-03-21T12:27:20.675 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: id: 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: services: 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.585773s) 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: data: 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: pgs: 2024-03-21T12:27:20.676 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:21 smithi172 ceph-mon[24470]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T12:27:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:21 smithi172 ceph-mon[24470]: monmap e1: 1 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} 2024-03-21T12:27:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:21 smithi172 ceph-mon[24470]: fsmap 2024-03-21T12:27:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:21 smithi172 ceph-mon[24470]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T12:27:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:21 smithi172 ceph-mon[24470]: mgrmap e1: no daemons active 2024-03-21T12:27:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:21 smithi172 ceph-mon[24470]: from='client.? 172.21.15.172:0/2483463210' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:27:21.760 INFO:teuthology.orchestra.run.smithi172.stderr:mon is available 2024-03-21T12:27:21.760 INFO:teuthology.orchestra.run.smithi172.stderr:Assimilating anything we can from ceph.conf... 2024-03-21T12:27:22.289 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: [global] 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: fsid = 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.172:3300,v1:172.21.15.172:6789] 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: [mgr] 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-21T12:27:22.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:22.291 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: [osd] 2024-03-21T12:27:22.291 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-21T12:27:22.291 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-21T12:27:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:22 smithi172 ceph-mon[24470]: from='client.? 172.21.15.172:0/3947696861' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T12:27:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:22 smithi172 ceph-mon[24470]: from='client.? 172.21.15.172:0/3947696861' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T12:27:22.849 INFO:teuthology.orchestra.run.smithi172.stderr:Generating new minimal ceph.conf... 2024-03-21T12:27:23.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:23 smithi172 ceph-mon[24470]: from='client.? 172.21.15.172:0/4120140772' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:27:24.906 INFO:teuthology.orchestra.run.smithi172.stderr:Restarting the monitor... 2024-03-21T12:27:25.277 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:24 smithi172 systemd[1]: Stopping Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:27:25.277 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:24 smithi172 bash[24688]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-21T12:27:25.277 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a[24465]: 2024-03-21T12:27:25.071+0000 7fc46ae59700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T12:27:25.277 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a[24465]: 2024-03-21T12:27:25.071+0000 7fc46ae59700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-21T12:27:25.574 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 podman[24699]: 2024-03-21 12:27:25.276948949 +0000 UTC m=+0.314942015 container died cbd78f8fd89c6a5b0a71b0d14318412b2407a326062e673429d0e95e1cb598e5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vendor=Red Hat, Inc.) 2024-03-21T12:27:25.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 podman[24699]: 2024-03-21 12:27:25.682105675 +0000 UTC m=+0.720098747 container cleanup cbd78f8fd89c6a5b0a71b0d14318412b2407a326062e673429d0e95e1cb598e5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, name=centos-stream, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, ceph=True, GIT_BRANCH=HEAD, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container) 2024-03-21T12:27:25.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 bash[24699]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a 2024-03-21T12:27:25.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 bash[24726]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon.a" found: no such container 2024-03-21T12:27:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 podman[24713]: 2024-03-21 12:27:25.857978263 +0000 UTC m=+0.584182465 container remove cbd78f8fd89c6a5b0a71b0d14318412b2407a326062e673429d0e95e1cb598e5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_CLEAN=True, release=754, architecture=x86_64, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, version=8, vcs-type=git, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-03-21T12:27:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T12:27:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 systemd[1]: Stopped Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:27:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:25 smithi172 systemd[1]: Starting Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:27:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 podman[24794]: 2024-03-21 12:27:25.99219129 +0000 UTC m=+0.019705063 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-21T12:27:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 podman[24794]: 2024-03-21 12:27:26.09190846 +0000 UTC m=+0.119422215 container create b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, com.redhat.component=centos-stream-container, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vcs-type=git, version=8, release=754, ceph=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8) 2024-03-21T12:27:26.405 INFO:teuthology.orchestra.run.smithi172.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-21T12:27:26.519 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 podman[24794]: 2024-03-21 12:27:26.319286632 +0000 UTC m=+0.346800387 container init b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., version=8, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-03-21T12:27:26.520 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 podman[24794]: 2024-03-21 12:27:26.322133772 +0000 UTC m=+0.349647537 container start b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, RELEASE=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754) 2024-03-21T12:27:26.520 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:27:26.520 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-21T12:27:26.520 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: pidfile_write: ignore empty --pid-file 2024-03-21T12:27:26.520 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: load: jerasure load: lrc 2024-03-21T12:27:26.520 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: RocksDB version: 6.15.5 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Compile date Apr 18 2022 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: DB SUMMARY 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: DB Session ID: TX1KOBEEVIETZTXRPRYK 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83166 ; 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:27:26.521 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.env: 0x55e766ac6860 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.fs: Posix File System 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.info_log: 0x55e767cd7dc0 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.statistics: (nil) 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.use_fsync: 0 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:27:26.522 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.db_log_dir: 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:27:26.523 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.write_buffer_manager: 0x55e767dc8240 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.unordered_write: 0 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:27:26.524 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.row_cache: None 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.wal_filter: None 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.preserve_deletes: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:27:26.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:27:26.526 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_open_files: -1 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Compression algorithms supported: 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kZSTD supported: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:27:26.527 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.merge_operator: 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_filter: None 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e767ca3d00) 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:27:26.528 INFO:journalctl@ceph.mon.a.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_type: 0 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: data_block_index_type: 0 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_shortening: 1 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: hash_index_allow_collision: 1 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: checksum: 1 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: no_block_cache: 0 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache: 0x55e767d0e170 2024-03-21T12:27:26.529 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_options: 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: capacity : 536870912 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: num_shard_bits : 4 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: strict_capacity_limit : 0 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_compressed: (nil) 2024-03-21T12:27:26.530 INFO:journalctl@ceph.mon.a.smithi172.stdout: persistent_cache: (nil) 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_size: 4096 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_size_deviation: 10 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_restart_interval: 16 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_block_restart_interval: 1 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: metadata_block_size: 4096 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: partition_filters: 0 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: use_delta_encoding: 1 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: whole_key_filtering: 1 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: verify_compression: 0 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: format_version: 4 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: enable_index_compression: 1 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_align: 0 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:27:26.531 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression: NoCompression 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.num_levels: 7 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:27:26.532 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:27:26.533 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.arena_block_size: 4194304 2024-03-21T12:27:26.534 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:27:26.535 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:27:26.536 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.ttl: 2592000 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711024046347494, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-21T12:27:26.537 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711024046348217, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79472, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77735, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10043, "raw_average_key_size": 47, "raw_value_size": 72031, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 212, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711024046, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "4a072e34-c1ac-47d3-bd13-f5d5fc2d2cc7", "db_session_id": "TX1KOBEEVIETZTXRPRYK"}} 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711024046348532, "job": 1, "event": "recovery_finished"} 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55e767cf4a80 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: DB pointer 0x55e767d04000 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** DB Stats ** 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:27:26.538 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: L0 2/0 79.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Sum 2/0 79.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.539 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative compaction: 0.00 GB write, 26.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval compaction: 0.00 GB write, 26.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:27:26.540 INFO:journalctl@ceph.mon.a.smithi172.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: L0 2/0 79.37 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: Sum 2/0 79.37 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.0 0.00 0.00 1 0.000 0 0 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:27:26.541 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative compaction: 0.00 GB write, 26.07 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: starting mon.a rank 0 at public addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] at bind addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???) e1 preinit fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).mds e1 new map 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).mds e1 print_map 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: e1 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T12:27:26.542 INFO:journalctl@ceph.mon.a.smithi172.stdout: legacy client fscid: -1 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout: No filesystems configured 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expand map: {default=false} 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta from 'false' to 'false' 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expanded map: {default=false} 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expand map: {default=info} 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta from 'info' to 'info' 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expanded map: {default=info} 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expand map: {default=daemon} 2024-03-21T12:27:26.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expanded map: {default=daemon} 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expand map: {default=debug} 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta from 'debug' to 'debug' 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: expand_channel_meta expanded map: {default=debug} 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 bash[24794]: b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 systemd[1]: Started Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: monmap e1: 1 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: fsmap 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T12:27:26.544 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:26 smithi172 ceph-mon[24813]: mgrmap e1: no daemons active 2024-03-21T12:27:27.511 INFO:teuthology.orchestra.run.smithi172.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-21T12:27:27.512 INFO:teuthology.orchestra.run.smithi172.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-21T12:27:27.512 INFO:teuthology.orchestra.run.smithi172.stderr:Creating mgr... 2024-03-21T12:27:27.512 INFO:teuthology.orchestra.run.smithi172.stderr:Verifying port 9283 ... 2024-03-21T12:27:27.681 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Failed to reset failed state of unit ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-21T12:27:27.688 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715.target.wants/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@.service. 2024-03-21T12:27:28.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:27 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/326969090' entity='client.admin' 2024-03-21T12:27:28.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 podman[25032]: 2024-03-21 12:27:28.059875409 +0000 UTC m=+0.119800544 container create 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, version=8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.component=centos-stream-container, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-21T12:27:28.399 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: disabled 2024-03-21T12:27:28.406 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: inactive 2024-03-21T12:27:28.406 INFO:teuthology.orchestra.run.smithi172.stderr:firewalld.service is not enabled 2024-03-21T12:27:28.407 INFO:teuthology.orchestra.run.smithi172.stderr:Not possible to enable service . firewalld.service is not available 2024-03-21T12:27:28.415 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: disabled 2024-03-21T12:27:28.422 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: inactive 2024-03-21T12:27:28.423 INFO:teuthology.orchestra.run.smithi172.stderr:firewalld.service is not enabled 2024-03-21T12:27:28.423 INFO:teuthology.orchestra.run.smithi172.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-21T12:27:28.423 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for mgr to start... 2024-03-21T12:27:28.423 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for mgr... 2024-03-21T12:27:28.549 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 podman[25032]: 2024-03-21 12:27:28.287175502 +0000 UTC m=+0.347100635 container init 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, architecture=x86_64, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8) 2024-03-21T12:27:28.550 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 podman[25032]: 2024-03-21 12:27:28.289623487 +0000 UTC m=+0.349548621 container start 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, version=8, io.openshift.tags=base centos centos-stream, ceph=True, maintainer=Guillaume Abrioux , vcs-type=git, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-21T12:27:28.550 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 bash[25032]: 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea 2024-03-21T12:27:28.550 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 systemd[1]: Started Ceph mgr.y for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:27:28.874 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:28.781+0000 7f6b5e48a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:27:28.969 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "fsid": "3bbfe794-e77e-11ee-95cd-87774f69a715", 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "health": { 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 0 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "a" 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T12:27:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T12:27:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "available": false, 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "iostat", 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "nfs", 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "restful" 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "services": {} 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "modified": "2024-03-21T12:27:20.089973+0000", 2024-03-21T12:27:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "services": {} 2024-03-21T12:27:28.974 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:28.974 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T12:27:28.974 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:27:29.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:28 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1256612598' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:27:29.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:28.874+0000 7f6b5e48a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:27:29.802 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:29 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:29.483+0000 7f6b5e48a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:27:29.802 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:29 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:29.571+0000 7f6b5e48a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:27:29.802 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:29 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:29.652+0000 7f6b5e48a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:27:30.076 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:29 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:29.801+0000 7f6b5e48a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:27:30.077 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:29 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:29.884+0000 7f6b5e48a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:27:30.077 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:29 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:29.993+0000 7f6b5e48a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:27:30.077 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:30.076+0000 7f6b5e48a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:27:30.196 INFO:teuthology.orchestra.run.smithi172.stderr:mgr not available, waiting (1/15)... 2024-03-21T12:27:30.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:30.242+0000 7f6b5e48a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:27:30.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:30.326+0000 7f6b5e48a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:27:31.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:31.366+0000 7f6b5e48a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:27:31.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:31.494+0000 7f6b5e48a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:27:32.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:31.682+0000 7f6b5e48a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:27:32.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:31.773+0000 7f6b5e48a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:27:32.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:31.912+0000 7f6b5e48a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:27:32.523 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:32.233+0000 7f6b5e48a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:27:32.523 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:32.459+0000 7f6b5e48a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:27:32.709 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:32.709 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:27:32.709 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "fsid": "3bbfe794-e77e-11ee-95cd-87774f69a715", 2024-03-21T12:27:32.709 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "health": { 2024-03-21T12:27:32.709 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 0 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "a" 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T12:27:32.710 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T12:27:32.711 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T12:27:32.714 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T12:27:32.715 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.715 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T12:27:32.716 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "available": false, 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "iostat", 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "nfs", 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "restful" 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "services": {} 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "modified": "2024-03-21T12:27:20.089973+0000", 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "services": {} 2024-03-21T12:27:32.717 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:32.718 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T12:27:32.718 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:27:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:32 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1741573605' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:27:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:32.544+0000 7f6b5e48a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:27:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:32.731+0000 7f6b5e48a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:27:33.286 INFO:teuthology.orchestra.run.smithi172.stderr:mgr not available, waiting (2/15)... 2024-03-21T12:27:33.951 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:33.661+0000 7f6b5e48a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:27:34.257 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:33.951+0000 7f6b5e48a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:27:34.257 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:34.045+0000 7f6b5e48a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:27:34.257 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:34.192+0000 7f6b5e48a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:27:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: Activating manager daemon y 2024-03-21T12:27:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: mgrmap e2: y(active, starting, since 0.00272685s) 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: Manager daemon y is now available 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' 2024-03-21T12:27:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:34 smithi172 ceph-mon[24813]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' 2024-03-21T12:27:35.929 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:35.929 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "fsid": "3bbfe794-e77e-11ee-95cd-87774f69a715", 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "health": { 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "checks": {}, 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mutes": [] 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum": [ 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 0 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "a" 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:35.930 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "monmap": { 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osdmap": { 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-21T12:27:35.931 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-21T12:27:35.934 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "pgmap": { 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.935 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "fsmap": { 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "available": true, 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "modules": [ 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "iostat", 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "nfs", 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "restful" 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ], 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "services": {} 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.936 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "servicemap": { 2024-03-21T12:27:35.937 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-21T12:27:35.937 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "modified": "2024-03-21T12:27:20.089973+0000", 2024-03-21T12:27:35.937 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "services": {} 2024-03-21T12:27:35.937 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: }, 2024-03-21T12:27:35.937 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-21T12:27:35.937 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:27:36.530 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:36 smithi172 ceph-mon[24813]: mgrmap e3: y(active, since 1.00684s) 2024-03-21T12:27:36.531 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:36 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2767816924' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:27:36.533 INFO:teuthology.orchestra.run.smithi172.stderr:mgr is available 2024-03-21T12:27:37.135 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:37.135 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: [global] 2024-03-21T12:27:37.135 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: fsid = 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:37.135 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: [mgr] 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: [osd] 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-21T12:27:37.136 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-21T12:27:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:37 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1537441400' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T12:27:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:37 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1537441400' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T12:27:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:37 smithi172 ceph-mon[24813]: mgrmap e4: y(active, since 3s) 2024-03-21T12:27:37.697 INFO:teuthology.orchestra.run.smithi172.stderr:Enabling cephadm module... 2024-03-21T12:27:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:38 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2143281046' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T12:27:38.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:38 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setuser ceph since I am not root 2024-03-21T12:27:38.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:38 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setgroup ceph since I am not root 2024-03-21T12:27:38.950 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:38 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:38.852+0000 7f1592e2a000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:27:38.951 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:38 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:38.942+0000 7f1592e2a000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:27:39.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2143281046' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T12:27:39.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-mon[24813]: mgrmap e5: y(active, since 4s) 2024-03-21T12:27:39.607 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:27:39.607 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-21T12:27:39.607 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "available": true, 2024-03-21T12:27:39.607 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-21T12:27:39.607 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-21T12:27:39.607 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:27:39.878 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:39.553+0000 7f1592e2a000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:27:39.878 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:39.647+0000 7f1592e2a000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:27:39.878 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:39.730+0000 7f1592e2a000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:27:39.878 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:39.877+0000 7f1592e2a000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:27:40.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:39.958+0000 7f1592e2a000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:27:40.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:40 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:40.065+0000 7f1592e2a000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:27:40.449 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:40 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1449118099' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T12:27:40.449 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:40 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:40.151+0000 7f1592e2a000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:27:40.449 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:40 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:40.314+0000 7f1592e2a000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:27:40.449 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:40 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:40.400+0000 7f1592e2a000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:27:40.451 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for the mgr to restart... 2024-03-21T12:27:40.451 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for mgr epoch 5... 2024-03-21T12:27:41.751 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:41.436+0000 7f1592e2a000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:27:41.751 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:41.564+0000 7f1592e2a000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:27:41.752 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:41.751+0000 7f1592e2a000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:27:42.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:41.837+0000 7f1592e2a000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:27:42.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:41.969+0000 7f1592e2a000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:27:42.585 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:42.286+0000 7f1592e2a000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:27:42.585 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:42.502+0000 7f1592e2a000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:27:42.586 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:42.585+0000 7f1592e2a000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:27:42.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:42.752+0000 7f1592e2a000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:27:43.941 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:43.655+0000 7f1592e2a000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:27:43.941 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:43.941+0000 7f1592e2a000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:27:44.247 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:44.034+0000 7f1592e2a000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:27:44.247 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:44.180+0000 7f1592e2a000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:27:44.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:44 smithi172 ceph-mon[24813]: Active manager daemon y restarted 2024-03-21T12:27:44.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:44 smithi172 ceph-mon[24813]: Activating manager daemon y 2024-03-21T12:27:44.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:44 smithi172 ceph-mon[24813]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T12:27:45.252 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:27:44] ENGINE Bus STARTING 2024-03-21T12:27:45.252 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:27:45] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:27:45.252 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:27:45] ENGINE Bus STARTED 2024-03-21T12:27:45.259 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:27:45.259 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-21T12:27:45.259 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "initialized": true 2024-03-21T12:27:45.259 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:27:45.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: mgrmap e6: y(active, starting, since 0.0522943s) 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: Manager daemon y is now available 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:45.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:45 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:27:45.832 INFO:teuthology.orchestra.run.smithi172.stderr:mgr epoch 5 is available 2024-03-21T12:27:45.832 INFO:teuthology.orchestra.run.smithi172.stderr:Setting orchestrator backend to cephadm... 2024-03-21T12:27:46.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:46 smithi172 ceph-mon[24813]: [21/Mar/2024:12:27:44] ENGINE Bus STARTING 2024-03-21T12:27:46.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:46 smithi172 ceph-mon[24813]: [21/Mar/2024:12:27:45] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:27:46.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:46 smithi172 ceph-mon[24813]: [21/Mar/2024:12:27:45] ENGINE Bus STARTED 2024-03-21T12:27:46.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:46 smithi172 ceph-mon[24813]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T12:27:46.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:46 smithi172 ceph-mon[24813]: mgrmap e7: y(active, since 1.05679s) 2024-03-21T12:27:46.464 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:46 smithi172 ceph-mon[24813]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T12:27:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:47 smithi172 ceph-mon[24813]: mgrmap e8: y(active, since 2s) 2024-03-21T12:27:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:47 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:47 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:27:47.538 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: value unchanged 2024-03-21T12:27:48.126 INFO:teuthology.orchestra.run.smithi172.stderr:Generating ssh key... 2024-03-21T12:27:48.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:48 smithi172 ceph-mon[24813]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-mon[24813]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:49.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: Generating public/private rsa key pair. 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: Your identification has been saved in /tmp/tmpukn95j41/key. 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: Your public key has been saved in /tmp/tmpukn95j41/key.pub. 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: The key fingerprint is: 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: SHA256:ahv03n+ZmnD+qDvRJ817ZIeIErCu907kfaCduwq4Frc ceph-3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: The key's randomart image is: 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: +---[RSA 3072]----+ 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | | 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | . | 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | o | 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | . . | 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | .. S..o + . | 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | .oo=.+oo+ = +| 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | .+=.=.=.oo *.| 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | ooE* ..* o= .| 2024-03-21T12:27:49.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: | ....o=.***+.. | 2024-03-21T12:27:49.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: +----[SHA256]-----+ 2024-03-21T12:27:50.309 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0CrDKVEYormW+wohLOmeouLZRsRz4Qn7a9WVEvlvgCcAoi/BXPGVNZ7slPljGgXBRG73VYVQA6ysn7tKbgTXSIjjuPtQ2vn2GKpTlkA5vufUS5/Re7BSNsOXfIR0pRID1PHUMz6nssPD+xw4S3TFPexyZNTIqEYjT1XnrNQxvAkkSkwWHWFXdOxTkoFK6+4IyPWeEVjjUVC3EfedBzXN2G45HbtB3hoII0OWAvOT7rM7DAXEzDUtKjyaMEqmUwDZ76F9Z1FexICcQ5V7weaTdz3eVnE/QIPjJY+qsrRE44OKPMFH1IMWXf6wSnBW+6M6Ar5z58u0dG/SPWnnCnAXRYmRsXJ8n+ZKej1RnOtJfBwRcJNbKHC8WIddnNSH5PnhFO9XI9Yw5a86w9aaq801M9zjsf1gPb6rjB/1jzqOUYO5FRJrZzBsFGS3eLHPoBEkdbqoIyYnnei1hjO1N6EcSo6+JGlqq4VDx4Jip5fPKzefyi9a2D+ImvaBahgCXKHU= ceph-3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:27:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:50 smithi172 ceph-mon[24813]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:50 smithi172 ceph-mon[24813]: Generating ssh key... 2024-03-21T12:27:50.822 INFO:teuthology.orchestra.run.smithi172.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-21T12:27:50.822 INFO:teuthology.orchestra.run.smithi172.stderr:Adding key to root@localhost authorized_keys... 2024-03-21T12:27:50.823 INFO:teuthology.orchestra.run.smithi172.stderr:Adding host smithi172... 2024-03-21T12:27:51.502 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:51 smithi172 ceph-mon[24813]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:52.435 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: Added host 'smithi172' with addr '172.21.15.172' 2024-03-21T12:27:52.735 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:52 smithi172 ceph-mon[24813]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "addr": "172.21.15.172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:52.735 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:52 smithi172 ceph-mon[24813]: Deploying cephadm binary to smithi172 2024-03-21T12:27:52.735 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:52 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:52.962 INFO:teuthology.orchestra.run.smithi172.stderr:Deploying unmanaged mon service... 2024-03-21T12:27:53.637 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-21T12:27:53.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:53 smithi172 ceph-mon[24813]: Added host smithi172 2024-03-21T12:27:53.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:53 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:27:54.445 INFO:teuthology.orchestra.run.smithi172.stderr:Deploying unmanaged mgr service... 2024-03-21T12:27:54.763 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:54 smithi172 ceph-mon[24813]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:54.764 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:54 smithi172 ceph-mon[24813]: Saving service mon spec with placement count:5 2024-03-21T12:27:54.764 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:54 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:55.056 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-21T12:27:56.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:56 smithi172 ceph-mon[24813]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:27:56.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:56 smithi172 ceph-mon[24813]: Saving service mgr spec with placement count:2 2024-03-21T12:27:56.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:56 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:57 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/597434971' entity='client.admin' 2024-03-21T12:27:57.950 INFO:teuthology.orchestra.run.smithi172.stderr:Enabling the dashboard module... 2024-03-21T12:27:58.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:58 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3207599998' entity='client.admin' 2024-03-21T12:27:59.764 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:59 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/4216818078' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T12:27:59.764 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:59 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:59.764 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:27:59 smithi172 ceph-mon[24813]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:27:59.764 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setuser ceph since I am not root 2024-03-21T12:27:59.765 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setgroup ceph since I am not root 2024-03-21T12:28:00.034 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:27:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:27:59.944+0000 7fc2f1788000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:28:00.034 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:00.032+0000 7fc2f1788000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:28:00.618 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/4216818078' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T12:28:00.618 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-mon[24813]: mgrmap e9: y(active, since 15s) 2024-03-21T12:28:00.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:00.638+0000 7fc2f1788000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:28:00.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:00.729+0000 7fc2f1788000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:28:00.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:00.813+0000 7fc2f1788000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:28:01.248 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:00.967+0000 7fc2f1788000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:28:01.248 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:01.050+0000 7fc2f1788000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:28:01.249 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:01.163+0000 7fc2f1788000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:28:01.249 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:01.247+0000 7fc2f1788000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:28:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:28:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-21T12:28:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "available": true, 2024-03-21T12:28:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-21T12:28:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-21T12:28:01.290 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:28:01.503 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:01 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1498917837' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T12:28:01.503 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:01.416+0000 7fc2f1788000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:28:01.503 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:01.501+0000 7fc2f1788000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:28:01.860 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for the mgr to restart... 2024-03-21T12:28:01.860 INFO:teuthology.orchestra.run.smithi172.stderr:Waiting for mgr epoch 9... 2024-03-21T12:28:02.869 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:02.550+0000 7fc2f1788000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:28:02.869 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:02.679+0000 7fc2f1788000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:28:02.869 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:02.867+0000 7fc2f1788000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:28:03.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:02.954+0000 7fc2f1788000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:28:03.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:03.089+0000 7fc2f1788000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:28:03.709 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:03.407+0000 7fc2f1788000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:28:03.709 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:03.625+0000 7fc2f1788000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:28:03.710 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:03.708+0000 7fc2f1788000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:28:04.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:03.877+0000 7fc2f1788000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:28:05.080 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:04 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:04.788+0000 7fc2f1788000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:28:05.394 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:05.079+0000 7fc2f1788000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:28:05.395 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:05.174+0000 7fc2f1788000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:28:05.395 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:28:05.324+0000 7fc2f1788000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:28:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:05 smithi172 ceph-mon[24813]: Active manager daemon y restarted 2024-03-21T12:28:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:05 smithi172 ceph-mon[24813]: Activating manager daemon y 2024-03-21T12:28:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:05 smithi172 ceph-mon[24813]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T12:28:06.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:28:06] ENGINE Bus STARTING 2024-03-21T12:28:06.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:28:06] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:28:06.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:28:06] ENGINE Bus STARTED 2024-03-21T12:28:06.412 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: { 2024-03-21T12:28:06.412 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-21T12:28:06.412 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: "initialized": true 2024-03-21T12:28:06.412 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: } 2024-03-21T12:28:06.725 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: mgrmap e10: y(active, starting, since 0.0524006s) 2024-03-21T12:28:06.725 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:06.725 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:28:06.725 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:28:06.725 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: Manager daemon y is now available 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:28:06.726 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:06.961 INFO:teuthology.orchestra.run.smithi172.stderr:mgr epoch 9 is available 2024-03-21T12:28:06.961 INFO:teuthology.orchestra.run.smithi172.stderr:Generating a dashboard self-signed certificate... 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:28:06] ENGINE Bus STARTING 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:28:06] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:28:06] ENGINE Bus STARTED 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: mgrmap e11: y(active, since 1.0672s) 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T12:28:07.586 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:07 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:07.691 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-21T12:28:08.268 INFO:teuthology.orchestra.run.smithi172.stderr:Creating initial admin user... 2024-03-21T12:28:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:08 smithi172 ceph-mon[24813]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:08 smithi172 ceph-mon[24813]: mgrmap e12: y(active, since 2s) 2024-03-21T12:28:09.115 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$yXJsNC8E7ToPHt9o.UhWl.z4z6qxiUJ31YUhvu9/gesfsuZ1jvq3a", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711024089, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-21T12:28:09.753 INFO:teuthology.orchestra.run.smithi172.stderr:Fetching dashboard port number... 2024-03-21T12:28:10.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:10 smithi172 ceph-mon[24813]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:10.340 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:10 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:10.340 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: 8443 2024-03-21T12:28:11.283 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:11 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1149984383' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T12:28:11.295 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: disabled 2024-03-21T12:28:11.302 INFO:teuthology.orchestra.run.smithi172.stderr:systemctl: inactive 2024-03-21T12:28:11.302 INFO:teuthology.orchestra.run.smithi172.stderr:firewalld.service is not enabled 2024-03-21T12:28:11.303 INFO:teuthology.orchestra.run.smithi172.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr:Ceph Dashboard is now available at: 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr: URL: https://smithi172.front.sepia.ceph.com:8443/ 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr: User: admin 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr: Password: 4jxx5o80e9 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:11.305 INFO:teuthology.orchestra.run.smithi172.stderr:Enabling autotune for osd_memory_target 2024-03-21T12:28:13.307 INFO:teuthology.orchestra.run.smithi172.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-21T12:28:13.621 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:13 smithi172 ceph-mon[24813]: mgrmap e13: y(active, since 7s) 2024-03-21T12:28:13.621 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:13 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3869062960' entity='client.admin' 2024-03-21T12:28:13.908 INFO:teuthology.orchestra.run.smithi172.stderr:You can access the Ceph CLI with: 2024-03-21T12:28:13.908 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:13.908 INFO:teuthology.orchestra.run.smithi172.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: ceph telemetry on 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr:For more information see: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T12:28:13.909 INFO:teuthology.orchestra.run.smithi172.stderr:Bootstrap complete. 2024-03-21T12:28:13.946 INFO:tasks.cephadm:Fetching config... 2024-03-21T12:28:13.946 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:13.946 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-21T12:28:13.962 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-21T12:28:13.962 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:13.962 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-21T12:28:14.018 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-21T12:28:14.018 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:14.018 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-21T12:28:14.085 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-21T12:28:14.086 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:14.086 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-21T12:28:14.141 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-21T12:28:14.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0CrDKVEYormW+wohLOmeouLZRsRz4Qn7a9WVEvlvgCcAoi/BXPGVNZ7slPljGgXBRG73VYVQA6ysn7tKbgTXSIjjuPtQ2vn2GKpTlkA5vufUS5/Re7BSNsOXfIR0pRID1PHUMz6nssPD+xw4S3TFPexyZNTIqEYjT1XnrNQxvAkkSkwWHWFXdOxTkoFK6+4IyPWeEVjjUVC3EfedBzXN2G45HbtB3hoII0OWAvOT7rM7DAXEzDUtKjyaMEqmUwDZ76F9Z1FexICcQ5V7weaTdz3eVnE/QIPjJY+qsrRE44OKPMFH1IMWXf6wSnBW+6M6Ar5z58u0dG/SPWnnCnAXRYmRsXJ8n+ZKej1RnOtJfBwRcJNbKHC8WIddnNSH5PnhFO9XI9Yw5a86w9aaq801M9zjsf1gPb6rjB/1jzqOUYO5FRJrZzBsFGS3eLHPoBEkdbqoIyYnnei1hjO1N6EcSo6+JGlqq4VDx4Jip5fPKzefyi9a2D+ImvaBahgCXKHU= ceph-3bbfe794-e77e-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T12:28:14.220 INFO:teuthology.orchestra.run.smithi172.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0CrDKVEYormW+wohLOmeouLZRsRz4Qn7a9WVEvlvgCcAoi/BXPGVNZ7slPljGgXBRG73VYVQA6ysn7tKbgTXSIjjuPtQ2vn2GKpTlkA5vufUS5/Re7BSNsOXfIR0pRID1PHUMz6nssPD+xw4S3TFPexyZNTIqEYjT1XnrNQxvAkkSkwWHWFXdOxTkoFK6+4IyPWeEVjjUVC3EfedBzXN2G45HbtB3hoII0OWAvOT7rM7DAXEzDUtKjyaMEqmUwDZ76F9Z1FexICcQ5V7weaTdz3eVnE/QIPjJY+qsrRE44OKPMFH1IMWXf6wSnBW+6M6Ar5z58u0dG/SPWnnCnAXRYmRsXJ8n+ZKej1RnOtJfBwRcJNbKHC8WIddnNSH5PnhFO9XI9Yw5a86w9aaq801M9zjsf1gPb6rjB/1jzqOUYO5FRJrZzBsFGS3eLHPoBEkdbqoIyYnnei1hjO1N6EcSo6+JGlqq4VDx4Jip5fPKzefyi9a2D+ImvaBahgCXKHU= ceph-3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:14.234 DEBUG:teuthology.orchestra.run.smithi177:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0CrDKVEYormW+wohLOmeouLZRsRz4Qn7a9WVEvlvgCcAoi/BXPGVNZ7slPljGgXBRG73VYVQA6ysn7tKbgTXSIjjuPtQ2vn2GKpTlkA5vufUS5/Re7BSNsOXfIR0pRID1PHUMz6nssPD+xw4S3TFPexyZNTIqEYjT1XnrNQxvAkkSkwWHWFXdOxTkoFK6+4IyPWeEVjjUVC3EfedBzXN2G45HbtB3hoII0OWAvOT7rM7DAXEzDUtKjyaMEqmUwDZ76F9Z1FexICcQ5V7weaTdz3eVnE/QIPjJY+qsrRE44OKPMFH1IMWXf6wSnBW+6M6Ar5z58u0dG/SPWnnCnAXRYmRsXJ8n+ZKej1RnOtJfBwRcJNbKHC8WIddnNSH5PnhFO9XI9Yw5a86w9aaq801M9zjsf1gPb6rjB/1jzqOUYO5FRJrZzBsFGS3eLHPoBEkdbqoIyYnnei1hjO1N6EcSo6+JGlqq4VDx4Jip5fPKzefyi9a2D+ImvaBahgCXKHU= ceph-3bbfe794-e77e-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T12:28:14.281 INFO:teuthology.orchestra.run.smithi177.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0CrDKVEYormW+wohLOmeouLZRsRz4Qn7a9WVEvlvgCcAoi/BXPGVNZ7slPljGgXBRG73VYVQA6ysn7tKbgTXSIjjuPtQ2vn2GKpTlkA5vufUS5/Re7BSNsOXfIR0pRID1PHUMz6nssPD+xw4S3TFPexyZNTIqEYjT1XnrNQxvAkkSkwWHWFXdOxTkoFK6+4IyPWeEVjjUVC3EfedBzXN2G45HbtB3hoII0OWAvOT7rM7DAXEzDUtKjyaMEqmUwDZ76F9Z1FexICcQ5V7weaTdz3eVnE/QIPjJY+qsrRE44OKPMFH1IMWXf6wSnBW+6M6Ar5z58u0dG/SPWnnCnAXRYmRsXJ8n+ZKej1RnOtJfBwRcJNbKHC8WIddnNSH5PnhFO9XI9Yw5a86w9aaq801M9zjsf1gPb6rjB/1jzqOUYO5FRJrZzBsFGS3eLHPoBEkdbqoIyYnnei1hjO1N6EcSo6+JGlqq4VDx4Jip5fPKzefyi9a2D+ImvaBahgCXKHU= ceph-3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:14.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-21T12:28:16.990 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-21T12:28:16.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-21T12:28:17.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:17 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/4215532416' entity='client.admin' 2024-03-21T12:28:17.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:17.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:28:17.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:17.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:17.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:19.469 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi177 2024-03-21T12:28:19.469 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:28:19.470 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.conf 2024-03-21T12:28:19.488 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:28:19.488 DEBUG:teuthology.orchestra.run.smithi177:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:19.545 INFO:tasks.cephadm:Adding host smithi177 to orchestrator... 2024-03-21T12:28:19.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch host add smithi177 2024-03-21T12:28:20.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:20.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:20.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:20.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:20.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:20.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:28:20.134 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:20.134 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:20.134 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:22.409 INFO:teuthology.orchestra.run.smithi172.stdout:Added host 'smithi177' with addr '172.21.15.177' 2024-03-21T12:28:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:22 smithi172 ceph-mon[24813]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:23.039 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: Deploying cephadm binary to smithi177 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: Added host smithi177 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:23.612 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:24.836 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:28:24.836 INFO:teuthology.orchestra.run.smithi172.stdout:[{"addr": "172.21.15.172", "hostname": "smithi172", "labels": [], "status": ""}, {"addr": "172.21.15.177", "hostname": "smithi177", "labels": [], "status": ""}] 2024-03-21T12:28:25.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:25 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:25.378 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-21T12:28:25.378 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-21T12:28:26.610 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:26 smithi172 ceph-mon[24813]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:28:26.610 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:26 smithi172 ceph-mon[24813]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:27.977 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2024-03-21T12:28:28.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:28.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:28.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:28:28.300 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:28.300 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:27 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2783502964' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T12:28:28.300 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:27 smithi172 ceph-mon[24813]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:28.615 INFO:tasks.cephadm:Adding mon.a on smithi172 2024-03-21T12:28:28.615 INFO:tasks.cephadm:Adding mon.c on smithi172 2024-03-21T12:28:28.615 INFO:tasks.cephadm:Adding mon.b on smithi177 2024-03-21T12:28:28.615 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b' 2024-03-21T12:28:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:28 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2783502964' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T12:28:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:28 smithi172 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:28:30.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:29 smithi172 ceph-mon[24813]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:30.720 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mon update... 2024-03-21T12:28:31.496 DEBUG:teuthology.orchestra.run.smithi172:mon.c> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.c.service 2024-03-21T12:28:31.499 DEBUG:teuthology.orchestra.run.smithi177:mon.b> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.b.service 2024-03-21T12:28:31.501 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:28:31.501 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:28:32.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:32.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: Saving service mon spec with placement smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b;count:3 2024-03-21T12:28:32.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:32.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:32.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:32.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: Deploying daemon mon.c on smithi172 2024-03-21T12:28:32.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:31 smithi172 ceph-mon[24813]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:33.415 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:28:33.415 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":1,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","modified":"2024-03-21T12:27:07.378991Z","created":"2024-03-21T12:27:07.378991Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-21T12:28:33.416 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 1 2024-03-21T12:28:33.765 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:33 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/3378581199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:33 smithi172 systemd[1]: Starting Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:28:34.450 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 podman[29199]: 2024-03-21 12:28:34.166054269 +0000 UTC m=+0.098482432 container create 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, release=754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, version=8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD) 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 podman[29199]: 2024-03-21 12:28:34.085802842 +0000 UTC m=+0.018231024 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 podman[29199]: 2024-03-21 12:28:34.33504596 +0000 UTC m=+0.267474128 container init 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8) 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 podman[29199]: 2024-03-21 12:28:34.33792506 +0000 UTC m=+0.270353219 container start 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , architecture=x86_64, vendor=Red Hat, Inc., release=754) 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: pidfile_write: ignore empty --pid-file 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: load: jerasure load: lrc 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: RocksDB version: 6.15.5 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Compile date Apr 18 2022 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: DB SUMMARY 2024-03-21T12:28:34.451 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: DB Session ID: Z4C3Z7FWBCC07TE2XIHW 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.env: 0x55a6b80c6860 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.fs: Posix File System 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.info_log: 0x55a6b90c1dc0 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.statistics: (nil) 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.use_fsync: 0 2024-03-21T12:28:34.452 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.db_log_dir: 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:28:34.453 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.write_buffer_manager: 0x55a6b91b2270 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:28:34.454 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.unordered_write: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.row_cache: None 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.wal_filter: None 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.preserve_deletes: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:28:34.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:28:34.456 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_open_files: -1 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Compression algorithms supported: 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kZSTD supported: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:28:34.457 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.merge_operator: 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_filter: None 2024-03-21T12:28:34.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a6b908dd00) 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_type: 0 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_index_type: 0 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_shortening: 1 2024-03-21T12:28:34.459 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: hash_index_allow_collision: 1 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: checksum: 1 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: no_block_cache: 0 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache: 0x55a6b90f8170 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_options: 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: capacity : 536870912 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_shard_bits : 4 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: strict_capacity_limit : 0 2024-03-21T12:28:34.460 INFO:journalctl@ceph.mon.c.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_compressed: (nil) 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: persistent_cache: (nil) 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size: 4096 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size_deviation: 10 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_restart_interval: 16 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_block_restart_interval: 1 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: metadata_block_size: 4096 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: partition_filters: 0 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: use_delta_encoding: 1 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-21T12:28:34.461 INFO:journalctl@ceph.mon.c.smithi172.stdout: whole_key_filtering: 1 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout: verify_compression: 0 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout: format_version: 4 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_index_compression: 1 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_align: 0 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression: NoCompression 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.num_levels: 7 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:28:34.462 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:28:34.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:28:34.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.arena_block_size: 4194304 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:28:34.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:28:34.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.ttl: 2592000 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T12:28:34.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711024114364399, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711024114364696, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711024114, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c64bf87f-3ec8-4c2c-ab05-2eb64893f026", "db_session_id": "Z4C3Z7FWBCC07TE2XIHW"}} 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711024114365021, "job": 1, "event": "recovery_finished"} 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55a6b90dea80 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: DB pointer 0x55a6b90ee000 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-21T12:28:34.469 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** DB Stats ** 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:28:34.470 INFO:journalctl@ceph.mon.c.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:28:34.471 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:28:34.472 INFO:journalctl@ceph.mon.c.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-21T12:28:34.473 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: using public_addrv [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: starting mon.c rank -1 at public addrs [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] at bind addrs [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(???) e0 preinit fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).mds e1 new map 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout: e1 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout: legacy client fscid: -1 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout: No filesystems configured 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T12:28:34.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mkfs 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: monmap e1: 1 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: fsmap 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e1: no daemons active 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2483463210' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3947696861' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3947696861' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T12:28:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/4120140772' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: monmap e1: 1 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: fsmap 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e1: no daemons active 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/326969090' entity='client.admin' 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1256612598' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1741573605' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Activating manager daemon y 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e2: y(active, starting, since 0.00272685s) 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:28:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14100 172.21.15.172:0/3411599493' entity='mgr.y' 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e3: y(active, since 1.00684s) 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2767816924' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1537441400' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1537441400' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e4: y(active, since 3s) 2024-03-21T12:28:34.477 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2143281046' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2143281046' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e5: y(active, since 4s) 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1449118099' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Active manager daemon y restarted 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Activating manager daemon y 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e6: y(active, starting, since 0.0522943s) 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.478 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: [21/Mar/2024:12:27:44] ENGINE Bus STARTING 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: [21/Mar/2024:12:27:45] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: [21/Mar/2024:12:27:45] ENGINE Bus STARTED 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e7: y(active, since 1.05679s) 2024-03-21T12:28:34.479 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e8: y(active, since 2s) 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Generating ssh key... 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "addr": "172.21.15.172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Deploying cephadm binary to smithi172 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.480 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Added host smithi172 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Saving service mon spec with placement count:5 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Saving service mgr spec with placement count:2 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/597434971' entity='client.admin' 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3207599998' entity='client.admin' 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/4216818078' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14118 172.21.15.172:0/1642380683' entity='mgr.y' 2024-03-21T12:28:34.481 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/4216818078' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e9: y(active, since 15s) 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1498917837' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Active manager daemon y restarted 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Activating manager daemon y 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e10: y(active, starting, since 0.0524006s) 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:28:34.482 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: [21/Mar/2024:12:28:06] ENGINE Bus STARTING 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: [21/Mar/2024:12:28:06] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: [21/Mar/2024:12:28:06] ENGINE Bus STARTED 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e11: y(active, since 1.0672s) 2024-03-21T12:28:34.483 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e12: y(active, since 2s) 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1149984383' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mgrmap e13: y(active, since 7s) 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3869062960' entity='client.admin' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/4215532416' entity='client.admin' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:28:34.484 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.485 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Deploying cephadm binary to smithi177 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Added host smithi177 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:34.486 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi177", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2783502964' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2783502964' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Saving service mon spec with placement smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b;count:3 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:34.487 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: Deploying daemon mon.c on smithi172 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/3378581199' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 ceph-mon[29218]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 bash[29199]: 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 2024-03-21T12:28:34.488 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:34 smithi172 systemd[1]: Started Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:28:35.000 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:28:35.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:28:39.789 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:28:39.789 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","modified":"2024-03-21T12:28:34.383163Z","created":"2024-03-21T12:27:07.378991Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3301","nonce":0},{"type":"v1","addr":"172.21.15.172:6790","nonce":0}]},"addr":"172.21.15.172:6790/0","public_addr":"172.21.15.172:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T12:28:39.791 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-03-21T12:28:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: mon.a calling monitor election 2024-03-21T12:28:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: mon.c calling monitor election 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: monmap e2: 2 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: fsmap 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: mgrmap e13: y(active, since 34s) 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: overall HEALTH_OK 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: mon.a calling monitor election 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: mon.c calling monitor election 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: monmap e2: 2 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: fsmap 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: mgrmap e13: y(active, since 34s) 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: overall HEALTH_OK 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:39.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[24813]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[24813]: Deploying daemon mon.b on smithi177 2024-03-21T12:28:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/3078844374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[29218]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[29218]: Deploying daemon mon.b on smithi177 2024-03-21T12:28:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/3078844374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:40 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:41.592 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:28:41.592 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:28:42.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:42 smithi172 ceph-mon[29218]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:42 smithi172 ceph-mon[24813]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:44.330 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:28:44.330 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":2,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","modified":"2024-03-21T12:28:34.383163Z","created":"2024-03-21T12:27:07.378991Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3301","nonce":0},{"type":"v1","addr":"172.21.15.172:6790","nonce":0}]},"addr":"172.21.15.172:6790/0","public_addr":"172.21.15.172:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T12:28:44.331 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 2 2024-03-21T12:28:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:44 smithi172 ceph-mon[24813]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:44 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/33178398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:44 smithi172 ceph-mon[29218]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:44 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/33178398' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:46.157 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T12:28:46.157 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T12:28:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: mon.a calling monitor election 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: mon.c calling monitor election 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: mon.b calling monitor election 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: fsmap 2024-03-21T12:28:50.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: mgrmap e13: y(active, since 44s) 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: overall HEALTH_OK 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: mon.a calling monitor election 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: mon.c calling monitor election 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: mon.b calling monitor election 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:28:50.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: fsmap 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: mgrmap e13: y(active, since 44s) 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: overall HEALTH_OK 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:50.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:51.083 INFO:teuthology.orchestra.run.smithi177.stdout: 2024-03-21T12:28:51.083 INFO:teuthology.orchestra.run.smithi177.stdout:{"epoch":3,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","modified":"2024-03-21T12:28:44.729353Z","created":"2024-03-21T12:27:07.378991Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3301","nonce":0},{"type":"v1","addr":"172.21.15.172:6790","nonce":0}]},"addr":"172.21.15.172:6790/0","public_addr":"172.21.15.172:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:3300","nonce":0},{"type":"v1","addr":"172.21.15.177:6789","nonce":0}]},"addr":"172.21.15.177:6789/0","public_addr":"172.21.15.177:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-21T12:28:51.085 INFO:teuthology.orchestra.run.smithi177.stderr:dumped monmap epoch 3 2024-03-21T12:28:51.554 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:51 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:51.576 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-21T12:28:51.576 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/889336912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/889336912' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T12:28:51.701 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.701 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:51.701 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:51.701 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:51.701 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:52.645 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: Reconfiguring daemon mon.a on smithi172 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: Reconfiguring daemon mon.a on smithi172 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:52 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: Reconfiguring daemon mon.a on smithi172 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:52.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:52 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:53.795 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:53 smithi172 ceph-mon[24813]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:28:53.795 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:53 smithi172 ceph-mon[24813]: Reconfiguring daemon mon.c on smithi172 2024-03-21T12:28:53.795 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:53 smithi172 ceph-mon[29218]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:28:53.796 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:53 smithi172 ceph-mon[29218]: Reconfiguring daemon mon.c on smithi172 2024-03-21T12:28:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:53 smithi177 ceph-mon[25767]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:28:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:53 smithi177 ceph-mon[25767]: Reconfiguring daemon mon.c on smithi172 2024-03-21T12:28:54.165 INFO:teuthology.orchestra.run.smithi172.stdout:# minimal ceph.conf for 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:54.166 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-03-21T12:28:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:28:54.166 INFO:teuthology.orchestra.run.smithi172.stdout: mon_host = [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] 2024-03-21T12:28:54.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:54 smithi172 ceph-mon[24813]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:54.700 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:54 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2842869145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:54.700 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:54 smithi172 ceph-mon[29218]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:54.701 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:54 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2842869145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:54.793 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:54 smithi177 ceph-mon[25767]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:54.793 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:54 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2842869145' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:55.411 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-21T12:28:55.412 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:55.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T12:28:55.442 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:55.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:55.510 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:28:55.510 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T12:28:55.539 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:28:55.539 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:28:55.605 INFO:tasks.cephadm:Adding mgr.y on smithi172 2024-03-21T12:28:55.605 INFO:tasks.cephadm:Adding mgr.x on smithi177 2024-03-21T12:28:55.606 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi172=y;smithi177=x' 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: Reconfiguring daemon mon.b on smithi177 2024-03-21T12:28:56.031 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:55 smithi177 ceph-mon[25767]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: Reconfiguring daemon mon.b on smithi177 2024-03-21T12:28:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[29218]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: Reconfiguring daemon mon.b on smithi177 2024-03-21T12:28:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:55 smithi172 ceph-mon[24813]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:57.193 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:57 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:57.193 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:57 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:57.193 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:57 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:57.194 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:57 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:57.194 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:57 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:57.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:57.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:57.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:57.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:57 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:57.568 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled mgr update... 2024-03-21T12:28:58.245 DEBUG:teuthology.orchestra.run.smithi177:mgr.x> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service 2024-03-21T12:28:58.247 INFO:tasks.cephadm:Deploying OSDs... 2024-03-21T12:28:58.247 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:28:58.248 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T12:28:58.267 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T12:28:58.267 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-21T12:28:58.324 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-21 12:28:33.913117681 +0000 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-21 12:26:00.970645819 +0000 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-21 12:26:00.970645819 +0000 2024-03-21T12:28:58.325 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-21 12:26:00.970645819 +0000 2024-03-21T12:28:58.325 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T12:28:58.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:28:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:58.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:58.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:58.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:58.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:58.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:58.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:58.392 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:28:58.393 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:28:58.394 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-21T12:28:58.394 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-21T12:28:58.394 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000204607 s, 2.5 MB/s 2024-03-21T12:28:58.395 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T12:28:58.452 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-21T12:28:58.509 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T12:28:58.509 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:58.509 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-21T12:28:58.510 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:58.510 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:58.510 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-21 12:28:33.913117681 +0000 2024-03-21T12:28:58.510 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-21 12:26:01.321637385 +0000 2024-03-21T12:28:58.510 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-21 12:26:01.321637385 +0000 2024-03-21T12:28:58.510 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-21 12:26:01.321637385 +0000 2024-03-21T12:28:58.510 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T12:28:58.578 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-21T12:28:58.578 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-21T12:28:58.578 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000194463 s, 2.6 MB/s 2024-03-21T12:28:58.579 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T12:28:58.635 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-21T12:28:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T12:28:58.691 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:58.691 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-21T12:28:58.691 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:58.691 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:58.692 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-21 12:28:33.914117657 +0000 2024-03-21T12:28:58.692 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-21 12:26:01.702628230 +0000 2024-03-21T12:28:58.692 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-21 12:26:01.702628230 +0000 2024-03-21T12:28:58.692 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-21 12:26:01.702628230 +0000 2024-03-21T12:28:58.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T12:28:58.758 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-21T12:28:58.758 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-21T12:28:58.758 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000207927 s, 2.5 MB/s 2024-03-21T12:28:58.759 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T12:28:58.818 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-21T12:28:58.874 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-21 12:28:33.914117657 +0000 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-21 12:26:02.056619724 +0000 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-21 12:26:02.056619724 +0000 2024-03-21T12:28:58.875 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-21 12:26:02.056619724 +0000 2024-03-21T12:28:58.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T12:28:58.940 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-21T12:28:58.941 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-21T12:28:58.941 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000209415 s, 2.4 MB/s 2024-03-21T12:28:58.942 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T12:28:58.997 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:28:58.997 DEBUG:teuthology.orchestra.run.smithi177:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T12:28:59.014 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T12:28:59.014 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_1 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-21 12:28:43.285852611 +0000 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-21 12:25:53.734138780 +0000 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-21 12:25:53.734138780 +0000 2024-03-21T12:28:59.069 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-21 12:25:53.734138780 +0000 2024-03-21T12:28:59.070 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T12:28:59.135 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-21T12:28:59.135 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-21T12:28:59.135 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000207517 s, 2.5 MB/s 2024-03-21T12:28:59.136 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T12:28:59.192 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_2 2024-03-21T12:28:59.247 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-21 12:28:43.286852583 +0000 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-21 12:25:54.074129325 +0000 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-21 12:25:54.074129325 +0000 2024-03-21T12:28:59.248 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-21 12:25:54.074129325 +0000 2024-03-21T12:28:59.248 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T12:28:59.311 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:59 smithi177 ceph-mon[25767]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi177=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:59.311 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:59 smithi177 ceph-mon[25767]: Saving service mgr spec with placement smithi172=y;smithi177=x;count:2 2024-03-21T12:28:59.311 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:28:59 smithi177 ceph-mon[25767]: Deploying daemon mgr.x on smithi177 2024-03-21T12:28:59.313 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-21T12:28:59.313 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-21T12:28:59.313 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000221459 s, 2.3 MB/s 2024-03-21T12:28:59.314 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T12:28:59.370 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_3 2024-03-21T12:28:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:59 smithi172 ceph-mon[29218]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi177=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:59 smithi172 ceph-mon[29218]: Saving service mgr spec with placement smithi172=y;smithi177=x;count:2 2024-03-21T12:28:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:28:59 smithi172 ceph-mon[29218]: Deploying daemon mgr.x on smithi177 2024-03-21T12:28:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:59 smithi172 ceph-mon[24813]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi177=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:28:59.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:59 smithi172 ceph-mon[24813]: Saving service mgr spec with placement smithi172=y;smithi177=x;count:2 2024-03-21T12:28:59.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:28:59 smithi172 ceph-mon[24813]: Deploying daemon mgr.x on smithi177 2024-03-21T12:28:59.424 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-21 12:28:43.286852583 +0000 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-21 12:25:54.410119982 +0000 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-21 12:25:54.410119982 +0000 2024-03-21T12:28:59.425 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-21 12:25:54.410119982 +0000 2024-03-21T12:28:59.425 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T12:28:59.566 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-21T12:28:59.566 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-21T12:28:59.566 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000187585 s, 2.7 MB/s 2024-03-21T12:28:59.567 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T12:28:59.582 DEBUG:teuthology.orchestra.run.smithi177:> stat /dev/vg_nvme/lv_4 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout:Access: 2024-03-21 12:28:59.525404656 +0000 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout:Modify: 2024-03-21 12:25:54.754110416 +0000 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout:Change: 2024-03-21 12:25:54.754110416 +0000 2024-03-21T12:28:59.640 INFO:teuthology.orchestra.run.smithi177.stdout: Birth: 2024-03-21 12:25:54.754110416 +0000 2024-03-21T12:28:59.641 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T12:28:59.743 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records in 2024-03-21T12:28:59.743 INFO:teuthology.orchestra.run.smithi177.stderr:1+0 records out 2024-03-21T12:28:59.743 INFO:teuthology.orchestra.run.smithi177.stderr:512 bytes copied, 0.000170701 s, 3.0 MB/s 2024-03-21T12:28:59.744 DEBUG:teuthology.orchestra.run.smithi177:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T12:28:59.759 INFO:tasks.cephadm:Deploying osd.0 on smithi172 with /dev/vg_nvme/lv_4... 2024-03-21T12:28:59.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T12:28:59.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:28:59 smithi177 systemd[1]: Starting Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:29:00.282 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:00 smithi177 ceph-mon[25767]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:00.283 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:28:59 smithi177 podman[26931]: 2024-03-21 12:28:59.879634705 +0000 UTC m=+0.017876365 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-21T12:29:00.283 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:28:59 smithi177 podman[26931]: 2024-03-21 12:28:59.989283446 +0000 UTC m=+0.127525143 container create ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, ceph=True, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-21T12:29:00.283 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 podman[26931]: 2024-03-21 12:29:00.183229689 +0000 UTC m=+0.321471355 container init ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vcs-type=git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.openshift.expose-services=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-21T12:29:00.283 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 podman[26931]: 2024-03-21 12:29:00.185769076 +0000 UTC m=+0.324010741 container start ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, version=8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, ceph=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-21T12:29:00.283 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 ceph-mgr[26950]: -- 172.21.15.177:0/2773367722 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560c6baf4340 con 0x560c6baccc00 2024-03-21T12:29:00.283 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 bash[26931]: ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 2024-03-21T12:29:00.301 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:00 smithi172 ceph-mon[24813]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:00.301 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:00 smithi172 ceph-mon[29218]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:00.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 systemd[1]: Started Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:29:01.057 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:00.708+0000 7f307e6bf000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:29:01.057 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:00.799+0000 7f307e6bf000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:01.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:01 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:01.414+0000 7f307e6bf000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:29:01.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:01.502+0000 7f307e6bf000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:29:01.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:01.584+0000 7f307e6bf000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:29:01.790 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T12:29:01.790 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T12:29:01.790 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-21T12:29:01.790 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-21T12:29:01.790 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116801 s, 898 MB/s 2024-03-21T12:29:01.790 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-21T12:29:02.003 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:01.732+0000 7f307e6bf000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:29:02.003 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:01.812+0000 7f307e6bf000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:29:02.003 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:01.920+0000 7f307e6bf000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:29:02.003 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:02.002+0000 7f307e6bf000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:29:02.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:02.162+0000 7f307e6bf000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:29:02.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:02.247+0000 7f307e6bf000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:29:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T12:29:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:29:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: Reconfiguring daemon mgr.y on smithi172 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[24813]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: Reconfiguring daemon mgr.y on smithi172 2024-03-21T12:29:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:02 smithi172 ceph-mon[29218]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: Reconfiguring daemon mgr.y on smithi172 2024-03-21T12:29:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:02 smithi177 ceph-mon[25767]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:02.874 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_4 2024-03-21T12:29:03.586 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:03.267+0000 7f307e6bf000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:29:03.587 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:03.397+0000 7f307e6bf000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:29:03.587 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:03.586+0000 7f307e6bf000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:29:03.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:03.672+0000 7f307e6bf000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:29:03.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:03.805+0000 7f307e6bf000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[24813]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:04 smithi172 ceph-mon[29218]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:04.424 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:04.424 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:04.424 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:04.425 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:04.425 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:04.425 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-mon[25767]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:04.425 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:04.122+0000 7f307e6bf000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:29:04.425 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:04.340+0000 7f307e6bf000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:29:04.426 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:04.423+0000 7f307e6bf000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:29:04.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:04.591+0000 7f307e6bf000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:29:05.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:05.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:05.543 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:05.544 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:05.544 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:05.544 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:05.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:05.497+0000 7f307e6bf000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:29:06.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:05.786+0000 7f307e6bf000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:29:06.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:05.878+0000 7f307e6bf000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:29:06.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:29:06.027+0000 7f307e6bf000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:29:06.322 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:06.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:06.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:29:06.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:29:06.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:29:06.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:29:06.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:29:06.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:29:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:29:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/1842279166' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/355943237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]: dispatch 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]: dispatch 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]': finished 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T12:29:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:07 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-21T12:29:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:29:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/355943237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]: dispatch 2024-03-21T12:29:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]: dispatch 2024-03-21T12:29:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]': finished 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: mgrmap e14: y(active, since 60s), standbys: x 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/355943237' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]: dispatch 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]: dispatch 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5"}]': finished 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T12:29:07.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:07 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:08 smithi177 ceph-mon[25767]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:08 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/191065826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:08 smithi172 ceph-mon[24813]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:08 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/191065826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:08 smithi172 ceph-mon[29218]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:08 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/191065826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:10 smithi177 ceph-mon[25767]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:10 smithi172 ceph-mon[24813]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:10 smithi172 ceph-mon[29218]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:12.582 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:12 smithi172 ceph-mon[24813]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:12.583 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:12 smithi172 ceph-mon[29218]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:12 smithi177 ceph-mon[25767]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:13.626 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:29:13.626 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:13.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:29:13.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:13.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:29:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:14.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:14 smithi172 ceph-mon[24813]: Deploying daemon osd.0 on smithi172 2024-03-21T12:29:14.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:14 smithi172 ceph-mon[24813]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:14 smithi172 ceph-mon[29218]: Deploying daemon osd.0 on smithi172 2024-03-21T12:29:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:14 smithi172 ceph-mon[29218]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:14 smithi177 ceph-mon[25767]: Deploying daemon osd.0 on smithi172 2024-03-21T12:29:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:14 smithi177 ceph-mon[25767]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:16.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:16 smithi172 ceph-mon[24813]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:16 smithi172 ceph-mon[29218]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:16.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:16 smithi177 ceph-mon[25767]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[24813]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[29218]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:18.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:18 smithi177 ceph-mon[25767]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:20.631 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:20 smithi172 ceph-mon[24813]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:20 smithi172 ceph-mon[29218]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:20 smithi177 ceph-mon[25767]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:21.377 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 0 on host 'smithi172' 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:21.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:21.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:21 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:21 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:21.982 DEBUG:teuthology.orchestra.run.smithi172:osd.0> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.0.service 2024-03-21T12:29:21.985 INFO:tasks.cephadm:Deploying osd.1 on smithi172 with /dev/vg_nvme/lv_3... 2024-03-21T12:29:21.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T12:29:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[29218]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[29218]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:29:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[29218]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T12:29:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[29218]: from='osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[29218]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[24813]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[24813]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:29:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[24813]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T12:29:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[24813]: from='osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-mon[24813]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:22.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:22 smithi177 ceph-mon[25767]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:22 smithi177 ceph-mon[25767]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:29:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:22 smithi177 ceph-mon[25767]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T12:29:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:22 smithi177 ceph-mon[25767]: from='osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:22 smithi177 ceph-mon[25767]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:23.126 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[32073]: 2024-03-21T12:29:22.824+0000 7f33e817a700 -1 osd.0 0 waiting for initial osdmap 2024-03-21T12:29:23.126 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:29:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[32073]: 2024-03-21T12:29:22.826+0000 7f33e0b0d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:29:24.098 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[24813]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[24813]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[24813]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[29218]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[29218]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:23 smithi172 ceph-mon[29218]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:24.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:23 smithi177 ceph-mon[25767]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:29:24.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:23 smithi177 ceph-mon[25767]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T12:29:24.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:23 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:23 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:23 smithi177 ceph-mon[25767]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T12:29:24.814 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T12:29:24.814 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T12:29:24.814 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-21T12:29:24.814 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-21T12:29:24.814 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122201 s, 858 MB/s 2024-03-21T12:29:24.814 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[29218]: osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413] boot 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[29218]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:29:24.980 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:29:24.981 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:24.981 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[24813]: osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413] boot 2024-03-21T12:29:24.981 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[24813]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T12:29:24.981 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:24 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:24 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:29:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:24 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:29:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:24 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:24 smithi177 ceph-mon[25767]: osd.0 [v2:172.21.15.172:6802/1201458413,v1:172.21.15.172:6803/1201458413] boot 2024-03-21T12:29:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:24 smithi177 ceph-mon[25767]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T12:29:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:24 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:29:25.640 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_3 2024-03-21T12:29:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:25 smithi177 ceph-mon[25767]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T12:29:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:25 smithi177 ceph-mon[25767]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:25 smithi172 ceph-mon[29218]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T12:29:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:25 smithi172 ceph-mon[29218]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:25 smithi172 ceph-mon[24813]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T12:29:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:25 smithi172 ceph-mon[24813]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:27.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi172 2024-03-21T12:29:27.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:27.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi172 to 16064M 2024-03-21T12:29:27.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi172 2024-03-21T12:29:27.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:27.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi172 to 16064M 2024-03-21T12:29:27.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:27 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:27 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi172 2024-03-21T12:29:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:27 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:27 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:27 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi172 to 16064M 2024-03-21T12:29:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:27 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:27 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:28.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[24813]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:28.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:28.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:28.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:28.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[29218]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:28.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:28.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:28.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:28.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:28 smithi177 ceph-mon[25767]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:29 smithi172 ceph-mon[24813]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:29 smithi172 ceph-mon[29218]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:29.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:29 smithi177 ceph-mon[25767]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:30.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:30 smithi177 ceph-mon[25767]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:30.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:30 smithi172 ceph-mon[24813]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:30 smithi172 ceph-mon[29218]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:31.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:31 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2648424606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b12aac0-ae0c-48a7-962f-8be8499212dd"}]: dispatch 2024-03-21T12:29:31.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:31 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2648424606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b12aac0-ae0c-48a7-962f-8be8499212dd"}]': finished 2024-03-21T12:29:31.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:31 smithi177 ceph-mon[25767]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T12:29:31.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:31 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:31.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:31 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3551372258' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2648424606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b12aac0-ae0c-48a7-962f-8be8499212dd"}]: dispatch 2024-03-21T12:29:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2648424606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b12aac0-ae0c-48a7-962f-8be8499212dd"}]': finished 2024-03-21T12:29:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[24813]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T12:29:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3551372258' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2648424606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b12aac0-ae0c-48a7-962f-8be8499212dd"}]: dispatch 2024-03-21T12:29:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2648424606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b12aac0-ae0c-48a7-962f-8be8499212dd"}]': finished 2024-03-21T12:29:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[29218]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T12:29:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:31.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:31 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3551372258' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:32 smithi172 ceph-mon[24813]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:32 smithi172 ceph-mon[29218]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:32 smithi177 ceph-mon[25767]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:34 smithi172 ceph-mon[24813]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:34.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:34 smithi172 ceph-mon[29218]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:34.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:34 smithi177 ceph-mon[25767]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:36 smithi172 ceph-mon[29218]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:29:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:36 smithi172 ceph-mon[24813]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:29:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:36 smithi177 ceph-mon[25767]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:29:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:37.744 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:37 smithi172 ceph-mon[24813]: Deploying daemon osd.1 on smithi172 2024-03-21T12:29:37.744 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:37 smithi172 ceph-mon[29218]: Deploying daemon osd.1 on smithi172 2024-03-21T12:29:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:37 smithi177 ceph-mon[25767]: Deploying daemon osd.1 on smithi172 2024-03-21T12:29:38.744 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:38 smithi172 ceph-mon[24813]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:38.745 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:38 smithi172 ceph-mon[29218]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:38 smithi177 ceph-mon[25767]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:40.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:40 smithi172 ceph-mon[24813]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:40.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:40 smithi172 ceph-mon[29218]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:40.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:40 smithi177 ceph-mon[25767]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:42.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:42 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:42.989 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:42 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:42.990 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:42 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:42.990 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:42 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:42.990 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:42 smithi177 ceph-mon[25767]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:42.991 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:42.991 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:42.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:42.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:42.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[24813]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:42.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:42.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:42.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:42.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:42.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:42 smithi172 ceph-mon[29218]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[24813]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[24813]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[29218]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:44.582 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:44 smithi172 ceph-mon[29218]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T12:29:44.610 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 1 on host 'smithi172' 2024-03-21T12:29:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:44 smithi177 ceph-mon[25767]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:44 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:44 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:44 smithi177 ceph-mon[25767]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T12:29:45.306 DEBUG:teuthology.orchestra.run.smithi172:osd.1> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.1.service 2024-03-21T12:29:45.308 INFO:tasks.cephadm:Deploying osd.2 on smithi172 with /dev/vg_nvme/lv_2... 2024-03-21T12:29:45.308 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:45.875 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:45 smithi177 ceph-mon[25767]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T12:29:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[29218]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:29:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-mon[24813]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:45.898 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[35034]: 2024-03-21T12:29:45.758+0000 7f6e181d2700 -1 osd.1 0 waiting for initial osdmap 2024-03-21T12:29:45.898 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:29:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[35034]: 2024-03-21T12:29:45.762+0000 7f6e11366700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:29:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[24813]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:29:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[24813]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T12:29:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:46.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[29218]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:29:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[29218]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T12:29:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:46 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:47.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:46 smithi177 ceph-mon[25767]: from='osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:29:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:46 smithi177 ceph-mon[25767]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T12:29:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:46 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:46 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.043 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746] boot 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[29218]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.044 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746] boot 2024-03-21T12:29:48.045 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T12:29:48.045 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.045 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:47 smithi172 ceph-mon[24813]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:48.107 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T12:29:48.107 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T12:29:48.108 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-21T12:29:48.108 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-21T12:29:48.108 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124748 s, 841 MB/s 2024-03-21T12:29:48.108 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-21T12:29:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:29:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:29:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: osd.1 [v2:172.21.15.172:6810/2577289746,v1:172.21.15.172:6811/2577289746] boot 2024-03-21T12:29:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T12:29:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:29:48.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:47 smithi177 ceph-mon[25767]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-21T12:29:48.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_2 2024-03-21T12:29:49.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:48 smithi177 ceph-mon[25767]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T12:29:49.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:48 smithi172 ceph-mon[24813]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T12:29:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:48 smithi172 ceph-mon[29218]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T12:29:50.744 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi172 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi172 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:50.745 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:50.746 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.746 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi172 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:29:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi172 to 8032M 2024-03-21T12:29:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi172 to 8032M 2024-03-21T12:29:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:51 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:51.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:51 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi172 to 8032M 2024-03-21T12:29:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:51 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:29:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:51 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:29:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:51 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:29:52.603 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:52 smithi172 ceph-mon[24813]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:52.603 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:52 smithi172 ceph-mon[24813]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:52.603 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:52 smithi172 ceph-mon[29218]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:52.603 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:52 smithi172 ceph-mon[29218]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:52 smithi177 ceph-mon[25767]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:29:52.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:52 smithi177 ceph-mon[25767]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:53 smithi177 ceph-mon[25767]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:53 smithi172 ceph-mon[24813]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:53 smithi172 ceph-mon[29218]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:54 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3213927213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4530bcd-a09e-4a0d-9903-209bbf2cc166"}]: dispatch 2024-03-21T12:29:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:54 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3213927213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4530bcd-a09e-4a0d-9903-209bbf2cc166"}]': finished 2024-03-21T12:29:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:54 smithi177 ceph-mon[25767]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T12:29:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:54 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:29:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:54 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/806561300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3213927213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4530bcd-a09e-4a0d-9903-209bbf2cc166"}]: dispatch 2024-03-21T12:29:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3213927213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4530bcd-a09e-4a0d-9903-209bbf2cc166"}]': finished 2024-03-21T12:29:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[24813]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T12:29:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:29:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/806561300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3213927213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4530bcd-a09e-4a0d-9903-209bbf2cc166"}]: dispatch 2024-03-21T12:29:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3213927213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4530bcd-a09e-4a0d-9903-209bbf2cc166"}]': finished 2024-03-21T12:29:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[29218]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T12:29:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:29:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:54 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/806561300' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:29:55.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:55 smithi177 ceph-mon[25767]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:55.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:55 smithi172 ceph-mon[29218]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:55.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:55 smithi172 ceph-mon[24813]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:58.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:29:58 smithi172 ceph-mon[24813]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:29:58 smithi172 ceph-mon[29218]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:29:58.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:29:58 smithi177 ceph-mon[25767]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[24813]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[24813]: overall HEALTH_OK 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[29218]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:00 smithi172 ceph-mon[29218]: overall HEALTH_OK 2024-03-21T12:30:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:00 smithi177 ceph-mon[25767]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:00 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:30:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:00 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:00 smithi177 ceph-mon[25767]: overall HEALTH_OK 2024-03-21T12:30:01.823 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:01 smithi172 ceph-mon[24813]: Deploying daemon osd.2 on smithi172 2024-03-21T12:30:01.823 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:01 smithi172 ceph-mon[29218]: Deploying daemon osd.2 on smithi172 2024-03-21T12:30:01.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:01 smithi177 ceph-mon[25767]: Deploying daemon osd.2 on smithi172 2024-03-21T12:30:02.641 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:02 smithi172 ceph-mon[24813]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:02.641 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:02 smithi172 ceph-mon[29218]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:02 smithi177 ceph-mon[25767]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:04.730 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:04 smithi172 ceph-mon[24813]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:04.731 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:04 smithi172 ceph-mon[29218]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:04.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:04 smithi177 ceph-mon[25767]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:30:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:06.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:30:06.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:30:08.090 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 2 on host 'smithi172' 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:08.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:08 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:08.716 DEBUG:teuthology.orchestra.run.smithi172:osd.2> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T12:30:08.718 INFO:tasks.cephadm:Deploying osd.3 on smithi172 with /dev/vg_nvme/lv_1... 2024-03-21T12:30:08.718 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:08 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:09.515 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[24813]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T12:30:09.515 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[24813]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T12:30:09.515 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:09.515 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[24813]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:09.515 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[29218]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T12:30:09.516 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[29218]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T12:30:09.516 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:09.516 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-mon[29218]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:09.516 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[37964]: 2024-03-21T12:30:09.413+0000 7f776a684700 -1 osd.2 0 waiting for initial osdmap 2024-03-21T12:30:09.516 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:09 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[37964]: 2024-03-21T12:30:09.416+0000 7f776481a700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:30:09.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:09 smithi177 ceph-mon[25767]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T12:30:09.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:09 smithi177 ceph-mon[25767]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T12:30:09.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:09 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:09.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:09 smithi177 ceph-mon[25767]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:10.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[29218]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:30:10.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[29218]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T12:30:10.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:10.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[29218]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:10.715 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:10.715 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[24813]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:30:10.715 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[24813]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T12:30:10.715 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:10.715 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[24813]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:10.715 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:10 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:10.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:10 smithi177 ceph-mon[25767]: from='osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:30:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:10 smithi177 ceph-mon[25767]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T12:30:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:10 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:10 smithi177 ceph-mon[25767]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:10 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T12:30:11.000 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T12:30:11.000 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-21T12:30:11.001 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-21T12:30:11.001 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126037 s, 832 MB/s 2024-03-21T12:30:11.001 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-21T12:30:11.729 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[24813]: osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223] boot 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[24813]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[29218]: osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223] boot 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[29218]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T12:30:11.730 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:11 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:11.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:11 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:30:11.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:11 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:30:11.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:11 smithi177 ceph-mon[25767]: osd.2 [v2:172.21.15.172:6818/387439223,v1:172.21.15.172:6819/387439223] boot 2024-03-21T12:30:11.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:11 smithi177 ceph-mon[25767]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T12:30:11.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:11 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:30:12.193 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_1 2024-03-21T12:30:12.688 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:12 smithi172 ceph-mon[29218]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:12.688 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:12 smithi172 ceph-mon[29218]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T12:30:12.688 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:12 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-21T12:30:12.688 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:12 smithi172 ceph-mon[24813]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:12.689 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:12 smithi172 ceph-mon[24813]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T12:30:12.689 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:12 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-21T12:30:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:12 smithi177 ceph-mon[25767]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-21T12:30:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:12 smithi177 ceph-mon[25767]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T12:30:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:12 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-21T12:30:13.464 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-21T12:30:13.464 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T12:30:13.464 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.465 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.466 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:14.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:30:13 smithi172 sudo[39253]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:30:14.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:30:13 smithi172 sudo[39253]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:14.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:30:13 smithi172 sudo[39253]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:14.315 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39274]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:30:14.315 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39274]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:14.316 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39274]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:14.590 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39281]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:30:14.590 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39281]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:14.590 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39281]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:14.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi172 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi172 to 5354M 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[24813]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[24813]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi172 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi172 to 5354M 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[29218]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[29218]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T12:30:14.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:14 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:14 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi172 2024-03-21T12:30:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:14 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi172 to 5354M 2024-03-21T12:30:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:14 smithi177 ceph-mon[25767]: pgmap v74: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:14 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T12:30:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:14 smithi177 ceph-mon[25767]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T12:30:14.759 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:14 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:14.844 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39306]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:30:14.844 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39306]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:14.844 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39306]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:15.107 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39346]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:30:15.108 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39346]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:15.108 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:14 smithi172 sudo[39346]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:15.361 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39391]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:30:15.361 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39391]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:15.361 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39391]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:15.619 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39412]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:30:15.619 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39412]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:15.619 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39412]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:15.871 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39418]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:30:15.871 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39418]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:15.871 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39418]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[24813]: osdmap e22: 3 total, 3 up, 3 in 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[24813]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[29218]: osdmap e22: 3 total, 3 up, 3 in 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[29218]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T12:30:15.872 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 ceph-mon[29218]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:15 smithi177 ceph-mon[25767]: osdmap e22: 3 total, 3 up, 3 in 2024-03-21T12:30:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:15 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:15 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:15 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:15 smithi177 ceph-mon[25767]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T12:30:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:15 smithi177 ceph-mon[25767]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:16.130 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39421]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:30:16.130 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39421]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:16.130 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:15 smithi172 sudo[39421]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 sudo[39443]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:30:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 sudo[39443]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 sudo[39443]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:16.621 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 sudo[27286]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T12:30:16.621 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 sudo[27286]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:16.621 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 sudo[27286]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:30:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:16.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:16.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 sudo[27289]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 sudo[27289]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 sudo[27289]: pam_unix(sudo:session): session closed for user root 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:17.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:17.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:17.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:17.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:30:17.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:30:17.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2109634616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781e720a-b7ed-47b2-9d17-5b2458d9d5f5"}]: dispatch 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2109634616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781e720a-b7ed-47b2-9d17-5b2458d9d5f5"}]': finished 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3961534787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2109634616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781e720a-b7ed-47b2-9d17-5b2458d9d5f5"}]: dispatch 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2109634616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781e720a-b7ed-47b2-9d17-5b2458d9d5f5"}]': finished 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:17 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3961534787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2109634616' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "781e720a-b7ed-47b2-9d17-5b2458d9d5f5"}]: dispatch 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2109634616' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "781e720a-b7ed-47b2-9d17-5b2458d9d5f5"}]': finished 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:17 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3961534787' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:30:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:20 smithi177 ceph-mon[25767]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:20.887 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:20 smithi172 ceph-mon[24813]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:20.887 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:20 smithi172 ceph-mon[29218]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:22 smithi172 ceph-mon[24813]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:22 smithi172 ceph-mon[29218]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:22 smithi177 ceph-mon[25767]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:23.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:30:23.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:23 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:23 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:30:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:23 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:23.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:23 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:30:23.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:23 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:24.674 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:24 smithi172 ceph-mon[24813]: Deploying daemon osd.3 on smithi172 2024-03-21T12:30:24.674 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:24 smithi172 ceph-mon[24813]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:24.675 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:24 smithi172 ceph-mon[29218]: Deploying daemon osd.3 on smithi172 2024-03-21T12:30:24.675 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:24 smithi172 ceph-mon[29218]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:24.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:24 smithi177 ceph-mon[25767]: Deploying daemon osd.3 on smithi172 2024-03-21T12:30:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:24 smithi177 ceph-mon[25767]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:26.718 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:26 smithi172 ceph-mon[24813]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:26.718 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:26 smithi172 ceph-mon[29218]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:26.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:26 smithi177 ceph-mon[25767]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[24813]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[29218]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:28 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:28.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:28 smithi177 ceph-mon[25767]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:28 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:30.742 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:30 smithi172 ceph-mon[24813]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:30.743 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:30 smithi172 ceph-mon[29218]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:30.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:30 smithi177 ceph-mon[25767]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:31.338 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 3 on host 'smithi172' 2024-03-21T12:30:31.935 DEBUG:teuthology.orchestra.run.smithi172:osd.3> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.3.service 2024-03-21T12:30:31.937 INFO:tasks.cephadm:Deploying osd.4 on smithi177 with /dev/vg_nvme/lv_4... 2024-03-21T12:30:31.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T12:30:32.505 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:32 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:32.506 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:32 smithi177 ceph-mon[25767]: from='osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:30:32.506 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:32 smithi177 ceph-mon[25767]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:30:32.506 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:32 smithi177 ceph-mon[25767]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:32.506 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:32 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:32.575 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:32.575 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[24813]: from='osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:30:32.575 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[24813]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:30:32.575 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[24813]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:32.575 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:32.576 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:32.576 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[29218]: from='osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:30:32.576 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[29218]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T12:30:32.576 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[29218]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:32.576 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:32 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:33.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:33 smithi177 ceph-mon[25767]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T12:30:33.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:33 smithi177 ceph-mon[25767]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T12:30:33.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:33 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:33.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:33 smithi177 ceph-mon[25767]: from='osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:33.384 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:33 smithi177 ceph-mon[25767]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[29218]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T12:30:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[29218]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[29218]: from='osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[29218]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[24813]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[24813]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[24813]: from='osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-mon[24813]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:30:33.647 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[40948]: 2024-03-21T12:30:33.355+0000 7f1366399700 -1 osd.3 0 waiting for initial osdmap 2024-03-21T12:30:33.648 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 12:30:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[40948]: 2024-03-21T12:30:33.358+0000 7f135f52d700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:30:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T12:30:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T12:30:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-21T12:30:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-21T12:30:33.862 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111356 s, 942 MB/s 2024-03-21T12:30:33.862 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-21T12:30:34.431 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:34 smithi177 ceph-mon[25767]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:30:34.431 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:34 smithi177 ceph-mon[25767]: osdmap e25: 4 total, 3 up, 4 in 2024-03-21T12:30:34.432 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:34 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:34.432 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:34 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:34.432 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:34 smithi177 ceph-mon[25767]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:34.454 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_4 2024-03-21T12:30:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[24813]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:30:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[24813]: osdmap e25: 4 total, 3 up, 4 in 2024-03-21T12:30:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[24813]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[29218]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[29218]: osdmap e25: 4 total, 3 up, 4 in 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:34.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:34 smithi172 ceph-mon[29218]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[24813]: osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542] boot 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[24813]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:30:35.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:35.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[29218]: osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542] boot 2024-03-21T12:30:35.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[29218]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T12:30:35.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:35 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:35.623 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:35 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:30:35.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:35 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:30:35.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:35 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:35.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:35 smithi177 ceph-mon[25767]: osd.3 [v2:172.21.15.172:6826/3588711542,v1:172.21.15.172:6827/3588711542] boot 2024-03-21T12:30:35.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:35 smithi177 ceph-mon[25767]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T12:30:35.624 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:35 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:36.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: osdmap e27: 4 total, 4 up, 4 in 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:36.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:36.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:37.763 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:37 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi172 2024-03-21T12:30:37.763 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:37 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi172 to 4016M 2024-03-21T12:30:37.763 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:37 smithi177 ceph-mon[25767]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:30:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:37 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi172 2024-03-21T12:30:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:37 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi172 to 4016M 2024-03-21T12:30:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:37 smithi172 ceph-mon[29218]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:30:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:37 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi172 2024-03-21T12:30:37.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:37 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi172 to 4016M 2024-03-21T12:30:37.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:37 smithi172 ceph-mon[24813]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:30:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:38 smithi177 ceph-mon[25767]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:38 smithi172 ceph-mon[24813]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:38 smithi172 ceph-mon[29218]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/3255976119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]: dispatch 2024-03-21T12:30:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]: dispatch 2024-03-21T12:30:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]': finished 2024-03-21T12:30:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:39 smithi177 ceph-mon[25767]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T12:30:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:39 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/3062859955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/3255976119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]: dispatch 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]: dispatch 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]': finished 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[29218]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/3062859955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/3255976119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]: dispatch 2024-03-21T12:30:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]: dispatch 2024-03-21T12:30:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d5995e0a-38a0-43ba-b9a5-981b861a32d1"}]': finished 2024-03-21T12:30:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[24813]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T12:30:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:39.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/3062859955' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:30:40.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:40 smithi172 ceph-mon[24813]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:40.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:40 smithi172 ceph-mon[29218]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:40.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:40 smithi177 ceph-mon[25767]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:42.723 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:42 smithi177 ceph-mon[25767]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:42 smithi172 ceph-mon[24813]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:42 smithi172 ceph-mon[29218]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:44.598 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:44 smithi177 ceph-mon[25767]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:44 smithi172 ceph-mon[24813]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:44 smithi172 ceph-mon[29218]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:45.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:30:45.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:30:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:30:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:46.595 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:46 smithi177 ceph-mon[25767]: Deploying daemon osd.4 on smithi177 2024-03-21T12:30:46.595 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:46 smithi177 ceph-mon[25767]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:46 smithi172 ceph-mon[24813]: Deploying daemon osd.4 on smithi177 2024-03-21T12:30:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:46 smithi172 ceph-mon[24813]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:46.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:46 smithi172 ceph-mon[29218]: Deploying daemon osd.4 on smithi177 2024-03-21T12:30:46.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:46 smithi172 ceph-mon[29218]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:48.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:48 smithi177 ceph-mon[25767]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:48 smithi172 ceph-mon[24813]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:48 smithi172 ceph-mon[29218]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:50.698 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[24813]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[29218]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:50.699 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:50 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:50.770 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:50.770 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:50 smithi177 ceph-mon[25767]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:50.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:50.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:50.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:50 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:52.421 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 4 on host 'smithi177' 2024-03-21T12:30:52.674 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:52 smithi177 ceph-mon[25767]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:52.855 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:52 smithi172 ceph-mon[24813]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:52.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:52 smithi172 ceph-mon[29218]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:52.980 DEBUG:teuthology.orchestra.run.smithi177:osd.4> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.4.service 2024-03-21T12:30:52.982 INFO:tasks.cephadm:Deploying osd.5 on smithi177 with /dev/vg_nvme/lv_3... 2024-03-21T12:30:52.983 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T12:30:53.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:53 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:53 smithi177 ceph-mon[25767]: from='osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:30:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:53 smithi177 ceph-mon[25767]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:30:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:53 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[24813]: from='osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[29218]: from='osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[29218]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T12:30:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:53 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:54.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-mon[25767]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:54.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-mon[25767]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T12:30:54.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-mon[25767]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T12:30:54.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:54.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-mon[25767]: from='osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:30:54.600 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-mon[25767]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:30:54.600 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[29147]: 2024-03-21T12:30:54.439+0000 7f3b8b025700 -1 osd.4 0 waiting for initial osdmap 2024-03-21T12:30:54.600 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 12:30:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[29147]: 2024-03-21T12:30:54.443+0000 7f3b879c0700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:30:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[24813]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T12:30:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[24813]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T12:30:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[24813]: from='osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[29218]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[29218]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[29218]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[29218]: from='osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:30:54.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:54 smithi172 ceph-mon[29218]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:30:54.898 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T12:30:54.898 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T12:30:54.898 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-21T12:30:54.898 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-21T12:30:54.898 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126063 s, 832 MB/s 2024-03-21T12:30:54.898 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-21T12:30:55.531 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:55 smithi177 ceph-mon[25767]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:30:55.532 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:55 smithi177 ceph-mon[25767]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T12:30:55.532 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:55.532 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:55.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_3 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[24813]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[29218]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[29218]: osdmap e30: 5 total, 4 up, 5 in 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:56.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:30:56.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:30:56.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:56.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564] boot 2024-03-21T12:30:56.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T12:30:56.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:56.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:56 smithi177 ceph-mon[25767]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T12:30:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:30:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:30:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564] boot 2024-03-21T12:30:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T12:30:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[24813]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: osd.4 [v2:172.21.15.177:6800/3207213564,v1:172.21.15.177:6801/3207213564] boot 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:30:56.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:56 smithi172 ceph-mon[29218]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T12:30:58.675 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T12:30:58.675 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:58.676 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:58 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:30:58.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:58 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:30:59.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:59 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi177 2024-03-21T12:30:59.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:59 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi177 to 17088M 2024-03-21T12:30:59.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:59 smithi177 ceph-mon[25767]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T12:30:59.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:59 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:59.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:59 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:59.698 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:30:59 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:59.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi177 2024-03-21T12:30:59.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi177 to 17088M 2024-03-21T12:30:59.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[24813]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi177 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi177 to 17088M 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[29218]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:30:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:30:59 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:00 smithi177 ceph-mon[25767]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:00 smithi177 ceph-mon[25767]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T12:31:00.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:00 smithi177 ceph-mon[25767]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-21T12:31:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:00 smithi172 ceph-mon[24813]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:00 smithi172 ceph-mon[24813]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T12:31:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:00 smithi172 ceph-mon[24813]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-21T12:31:00.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:00 smithi172 ceph-mon[29218]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:00.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:00 smithi172 ceph-mon[29218]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T12:31:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:00 smithi172 ceph-mon[29218]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-21T12:31:01.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:01 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/3565221169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]: dispatch 2024-03-21T12:31:01.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:01 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]: dispatch 2024-03-21T12:31:01.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:01 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]': finished 2024-03-21T12:31:01.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:01 smithi177 ceph-mon[25767]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T12:31:01.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:01 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/3565221169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]: dispatch 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]: dispatch 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]': finished 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[24813]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/3565221169' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]: dispatch 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]: dispatch 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b82613bd-8471-4213-8c7d-e5a9ecde1844"}]': finished 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[29218]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T12:31:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:01 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:02 smithi172 ceph-mon[29218]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T12:31:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:02 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/2613060261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:02 smithi172 ceph-mon[24813]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T12:31:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:02 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/2613060261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:02 smithi177 ceph-mon[25767]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-21T12:31:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:02 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/2613060261' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:04.705 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:04 smithi177 ceph-mon[25767]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:04 smithi172 ceph-mon[24813]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:04 smithi172 ceph-mon[29218]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:06 smithi177 ceph-mon[25767]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:31:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:31:06.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:06 smithi172 ceph-mon[24813]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:06.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:31:06.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:31:06.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:06 smithi172 ceph-mon[29218]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:06.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:31:06.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:31:07.667 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:07 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:31:07.668 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:07 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:07.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:07 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:31:07.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:07 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:07.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:07 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:31:07.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:07 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:08.798 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:08 smithi177 ceph-mon[25767]: Deploying daemon osd.5 on smithi177 2024-03-21T12:31:08.798 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:08 smithi177 ceph-mon[25767]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:08 smithi172 ceph-mon[24813]: Deploying daemon osd.5 on smithi177 2024-03-21T12:31:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:08 smithi172 ceph-mon[24813]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:08.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:08 smithi172 ceph-mon[29218]: Deploying daemon osd.5 on smithi177 2024-03-21T12:31:08.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:08 smithi172 ceph-mon[29218]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:10.767 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:10 smithi177 ceph-mon[25767]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:10 smithi172 ceph-mon[29218]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:10 smithi172 ceph-mon[24813]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:12.783 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:12 smithi177 ceph-mon[25767]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:12.783 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:12 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:12 smithi172 ceph-mon[24813]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:12 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:12 smithi172 ceph-mon[29218]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:12 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:13.687 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:13.687 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:13.687 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:13 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:13 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:13 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:14.826 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:14 smithi177 ceph-mon[25767]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:14 smithi172 ceph-mon[24813]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:14 smithi172 ceph-mon[29218]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:15.451 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 5 on host 'smithi177' 2024-03-21T12:31:16.056 DEBUG:teuthology.orchestra.run.smithi177:osd.5> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.5.service 2024-03-21T12:31:16.058 INFO:tasks.cephadm:Deploying osd.6 on smithi177 with /dev/vg_nvme/lv_2... 2024-03-21T12:31:16.059 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T12:31:16.696 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:16 smithi177 ceph-mon[25767]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:16.696 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:16.696 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:16 smithi177 ceph-mon[25767]: from='osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:31:16.697 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:16 smithi177 ceph-mon[25767]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:31:16.697 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:16 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[24813]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[24813]: from='osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:31:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:31:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[29218]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[29218]: from='osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:31:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[29218]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T12:31:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:16 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:17.750 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-mon[25767]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T12:31:17.750 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-mon[25767]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T12:31:17.750 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:17.750 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-mon[25767]: from='osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:17.750 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-mon[25767]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:17.750 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[32089]: 2024-03-21T12:31:17.491+0000 7f68c66ad700 -1 osd.5 0 waiting for initial osdmap 2024-03-21T12:31:17.750 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 12:31:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[32089]: 2024-03-21T12:31:17.496+0000 7f68bf040700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[24813]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[24813]: from='osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[29218]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T12:31:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[29218]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T12:31:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[29218]: from='osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:17 smithi172 ceph-mon[29218]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:18.429 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T12:31:18.429 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T12:31:18.429 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-21T12:31:18.429 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-21T12:31:18.429 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0127525 s, 822 MB/s 2024-03-21T12:31:18.430 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-21T12:31:18.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:18 smithi177 ceph-mon[25767]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:18 smithi177 ceph-mon[25767]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:31:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:18 smithi177 ceph-mon[25767]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T12:31:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[24813]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[24813]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[29218]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:18.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[29218]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:31:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[29218]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T12:31:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:19.123 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_2 2024-03-21T12:31:19.763 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:19 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:31:19.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:19 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:31:19.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:19 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:19.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:19 smithi177 ceph-mon[25767]: osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767] boot 2024-03-21T12:31:19.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:19 smithi177 ceph-mon[25767]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T12:31:19.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:19 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[24813]: osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767] boot 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[24813]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:19.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:31:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:31:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[29218]: osd.5 [v2:172.21.15.177:6808/890621767,v1:172.21.15.177:6809/890621767] boot 2024-03-21T12:31:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[29218]: osdmap e39: 6 total, 6 up, 6 in 2024-03-21T12:31:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:19 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:31:20.797 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:20 smithi172 ceph-mon[24813]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:20.797 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:20 smithi172 ceph-mon[24813]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T12:31:20.797 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:20 smithi172 ceph-mon[29218]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:20.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:20 smithi172 ceph-mon[29218]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T12:31:20.806 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:20 smithi177 ceph-mon[25767]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-21T12:31:20.806 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:20 smithi177 ceph-mon[25767]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T12:31:21.823 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:21 smithi177 ceph-mon[25767]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T12:31:21.823 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:21 smithi177 ceph-mon[25767]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T12:31:21.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:21 smithi172 ceph-mon[29218]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T12:31:21.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:21 smithi172 ceph-mon[29218]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T12:31:21.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:21 smithi172 ceph-mon[24813]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T12:31:21.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:21 smithi172 ceph-mon[24813]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T12:31:22.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:22.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:31:22.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:31:22.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:22.741 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:22 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T12:31:22.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:22.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:22 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:23.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:23 smithi177 ceph-mon[25767]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:23.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:23 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi177 2024-03-21T12:31:23.786 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:23 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi177 to 8544M 2024-03-21T12:31:23.787 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:23 smithi177 ceph-mon[25767]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[24813]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi177 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi177 to 8544M 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[24813]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[29218]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi177 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi177 to 8544M 2024-03-21T12:31:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:23 smithi172 ceph-mon[29218]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:24 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/1580808098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]: dispatch 2024-03-21T12:31:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:24 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]: dispatch 2024-03-21T12:31:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:24 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]': finished 2024-03-21T12:31:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:24 smithi177 ceph-mon[25767]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T12:31:24.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:24 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/1580808098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]: dispatch 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]: dispatch 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]': finished 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[24813]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/1580808098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]: dispatch 2024-03-21T12:31:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]: dispatch 2024-03-21T12:31:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "761832a5-706c-4850-b314-f7a0419c729b"}]': finished 2024-03-21T12:31:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[29218]: osdmap e44: 7 total, 6 up, 7 in 2024-03-21T12:31:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:24 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:25 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/4105254402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:25 smithi177 ceph-mon[25767]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T12:31:25.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:25 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/4105254402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:25.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:25 smithi172 ceph-mon[24813]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T12:31:25.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:25 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/4105254402' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:25.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:25 smithi172 ceph-mon[29218]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-21T12:31:26.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:26 smithi177 ceph-mon[25767]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T12:31:26.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:26 smithi177 ceph-mon[25767]: Cluster is now healthy 2024-03-21T12:31:26.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:26 smithi172 ceph-mon[24813]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T12:31:26.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:26 smithi172 ceph-mon[24813]: Cluster is now healthy 2024-03-21T12:31:26.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:26 smithi172 ceph-mon[29218]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-21T12:31:26.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:26 smithi172 ceph-mon[29218]: Cluster is now healthy 2024-03-21T12:31:27.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:27 smithi177 ceph-mon[25767]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:27.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:27 smithi172 ceph-mon[24813]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:27.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:27 smithi172 ceph-mon[29218]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:30.743 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:30 smithi177 ceph-mon[25767]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:31:30.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:30 smithi172 ceph-mon[24813]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:31:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:30 smithi172 ceph-mon[29218]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:31:31.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:31 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:31:31.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:31 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:31:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:31 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:31 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:31:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:31 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:32.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:32 smithi177 ceph-mon[25767]: Deploying daemon osd.6 on smithi177 2024-03-21T12:31:32.785 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:32 smithi177 ceph-mon[25767]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-21T12:31:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:32 smithi172 ceph-mon[24813]: Deploying daemon osd.6 on smithi177 2024-03-21T12:31:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:32 smithi172 ceph-mon[24813]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-21T12:31:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:32 smithi172 ceph-mon[29218]: Deploying daemon osd.6 on smithi177 2024-03-21T12:31:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:32 smithi172 ceph-mon[29218]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-21T12:31:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:34 smithi177 ceph-mon[25767]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T12:31:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:34 smithi172 ceph-mon[24813]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T12:31:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:34 smithi172 ceph-mon[29218]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T12:31:36.563 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:36 smithi177 ceph-mon[25767]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:36.563 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:36.563 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:36.563 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:36.563 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:36 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[29218]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[24813]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:36.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:36.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:36.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:36.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:36 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:38.819 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:38 smithi177 ceph-mon[25767]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:38 smithi172 ceph-mon[24813]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:38 smithi172 ceph-mon[29218]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:39.141 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 6 on host 'smithi177' 2024-03-21T12:31:39.730 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:39 smithi177 ceph-mon[25767]: from='osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:31:39.730 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:39 smithi177 ceph-mon[25767]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:31:39.730 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:39 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:39.753 DEBUG:teuthology.orchestra.run.smithi177:osd.6> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T12:31:39.755 INFO:tasks.cephadm:Deploying osd.7 on smithi177 with /dev/vg_nvme/lv_1... 2024-03-21T12:31:39.755 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T12:31:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:39 smithi172 ceph-mon[29218]: from='osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:31:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:39 smithi172 ceph-mon[29218]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:31:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:39 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:39 smithi172 ceph-mon[24813]: from='osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:31:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:39 smithi172 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T12:31:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:39 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: from='osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:40.608 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[34984]: 2024-03-21T12:31:40.506+0000 7fd69b485700 -1 osd.6 0 waiting for initial osdmap 2024-03-21T12:31:40.608 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 12:31:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[34984]: 2024-03-21T12:31:40.512+0000 7fd695e1c700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: from='osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: from='osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:31:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:40 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:41.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:31:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T12:31:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831] boot 2024-03-21T12:31:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T12:31:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:41 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.827 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T12:31:41.827 INFO:teuthology.orchestra.run.smithi177.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T12:31:41.827 INFO:teuthology.orchestra.run.smithi177.stderr: stderr: 10+0 records in 2024-03-21T12:31:41.828 INFO:teuthology.orchestra.run.smithi177.stderr:10+0 records out 2024-03-21T12:31:41.828 INFO:teuthology.orchestra.run.smithi177.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124853 s, 840 MB/s 2024-03-21T12:31:41.828 INFO:teuthology.orchestra.run.smithi177.stderr:--> Zapping successful for: 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831] boot 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:31:41.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T12:31:41.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:41.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: osd.6 [v2:172.21.15.177:6816/3923649831,v1:172.21.15.177:6817/3923649831] boot 2024-03-21T12:31:41.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: osdmap e47: 7 total, 7 up, 7 in 2024-03-21T12:31:41.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:41 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:31:42.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:42 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:31:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:42 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:31:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:42 smithi177 ceph-mon[25767]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:42 smithi177 ceph-mon[25767]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[24813]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[24813]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:31:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[29218]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-21T12:31:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:42 smithi172 ceph-mon[29218]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T12:31:42.967 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi177:vg_nvme/lv_1 2024-03-21T12:31:44.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:44 smithi177 ceph-mon[25767]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:44.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:44 smithi177 ceph-mon[25767]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T12:31:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:44 smithi172 ceph-mon[29218]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:44 smithi172 ceph-mon[29218]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T12:31:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:44 smithi172 ceph-mon[24813]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:44 smithi172 ceph-mon[24813]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T12:31:45.752 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T12:31:45.752 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.752 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.752 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.752 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.752 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.753 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.753 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:45.753 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:45.753 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:45.753 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:45 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T12:31:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:31:45.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:31:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:31:45.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:45 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi177 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi177 to 5696M 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:31:46.678 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:46 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi177 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi177 to 5696M 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:31:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi177 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi177 to 5696M 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T12:31:46.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:46 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:47.578 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:47 smithi177 ceph-mon[25767]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:47.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:47 smithi172 ceph-mon[24813]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:47.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:47 smithi172 ceph-mon[29218]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi177:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:31:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:48 smithi177 ceph-mon[25767]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T12:31:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:48 smithi172 ceph-mon[24813]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T12:31:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:48 smithi172 ceph-mon[29218]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T12:31:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:49 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/960351546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]: dispatch 2024-03-21T12:31:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:49 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]: dispatch 2024-03-21T12:31:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:49 smithi177 ceph-mon[25767]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]': finished 2024-03-21T12:31:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:49 smithi177 ceph-mon[25767]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T12:31:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:49 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:31:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:49 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/377202837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/960351546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]': finished 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[24813]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/377202837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/960351546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]: dispatch 2024-03-21T12:31:49.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[29218]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc315a72-89dd-4a44-8b88-efbe5562d78b"}]': finished 2024-03-21T12:31:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[29218]: osdmap e52: 8 total, 7 up, 8 in 2024-03-21T12:31:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:31:49.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:49 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/377202837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T12:31:50.797 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:50 smithi172 ceph-mon[24813]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:50.798 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:50 smithi172 ceph-mon[29218]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:50.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:50 smithi177 ceph-mon[25767]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:31:52.714 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:52 smithi177 ceph-mon[25767]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-21T12:31:52.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:52 smithi172 ceph-mon[24813]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-21T12:31:52.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:52 smithi172 ceph-mon[29218]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-21T12:31:54.833 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:54 smithi177 ceph-mon[25767]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:31:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:54 smithi172 ceph-mon[24813]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:31:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:54 smithi172 ceph-mon[29218]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:31:55.569 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:31:55.569 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:55 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:31:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:55 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:31:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:55 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:31:56.836 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:56 smithi177 ceph-mon[25767]: Deploying daemon osd.7 on smithi177 2024-03-21T12:31:56.836 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:56 smithi177 ceph-mon[25767]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T12:31:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:56 smithi172 ceph-mon[24813]: Deploying daemon osd.7 on smithi177 2024-03-21T12:31:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:56 smithi172 ceph-mon[24813]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T12:31:56.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:56 smithi172 ceph-mon[29218]: Deploying daemon osd.7 on smithi177 2024-03-21T12:31:56.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:56 smithi172 ceph-mon[29218]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-21T12:31:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:31:58 smithi177 ceph-mon[25767]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:58.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:31:58 smithi172 ceph-mon[24813]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:31:58.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:31:58 smithi172 ceph-mon[29218]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:00 smithi177 ceph-mon[25767]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:00 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:00 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:00 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:00 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[24813]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:00.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[29218]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:00 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:02.605 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:02 smithi177 ceph-mon[25767]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:02.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:02 smithi172 ceph-mon[24813]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:02 smithi172 ceph-mon[29218]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:03.318 INFO:teuthology.orchestra.run.smithi177.stdout:Created osd(s) 7 on host 'smithi177' 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:03 smithi177 ceph-mon[25767]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:32:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:32:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[24813]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:03 smithi172 ceph-mon[29218]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:03.945 DEBUG:teuthology.orchestra.run.smithi177:osd.7> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.7.service 2024-03-21T12:32:03.947 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-21T12:32:03.947 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T12:32:05.179 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-mon[25767]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T12:32:05.180 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-mon[25767]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T12:32:05.180 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:05.180 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-mon[25767]: from='osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:32:05.180 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-mon[25767]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:32:05.180 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[37914]: 2024-03-21T12:32:04.976+0000 7f0dd5474700 -1 osd.7 0 waiting for initial osdmap 2024-03-21T12:32:05.180 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 12:32:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[37914]: 2024-03-21T12:32:04.985+0000 7f0dce608700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[24813]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[24813]: from='osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[29218]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[29218]: osdmap e53: 8 total, 7 up, 8 in 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[29218]: from='osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:32:05.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:04 smithi172 ceph-mon[29218]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T12:32:05.948 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3491305057' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781] boot 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T12:32:06.231 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:05 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:06.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3491305057' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781] boot 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]': finished 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3491305057' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:32:06.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: osd.7 [v2:172.21.15.177:6824/3906799781,v1:172.21.15.177:6825/3906799781] boot 2024-03-21T12:32:06.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: osdmap e55: 8 total, 8 up, 8 in 2024-03-21T12:32:06.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:05 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:06.712 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1711024301,"num_in_osds":8,"osd_in_since":1711024308,"num_remapped_pgs":0} 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: purged_snaps scrub starts 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: purged_snaps scrub ok 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:06 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: purged_snaps scrub starts 2024-03-21T12:32:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: purged_snaps scrub ok 2024-03-21T12:32:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: purged_snaps scrub starts 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: purged_snaps scrub ok 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: osdmap e56: 8 total, 8 up, 8 in 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:32:07.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:06 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:07.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T12:32:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi177 2024-03-21T12:32:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi177 to 4272M 2024-03-21T12:32:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[24813]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[24813]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T12:32:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi177 2024-03-21T12:32:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi177 to 4272M 2024-03-21T12:32:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[29218]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:08 smithi172 ceph-mon[29218]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T12:32:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:08 smithi177 ceph-mon[25767]: Detected new or changed devices on smithi177 2024-03-21T12:32:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:08 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi177 to 4272M 2024-03-21T12:32:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:08 smithi177 ceph-mon[25767]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:08 smithi177 ceph-mon[25767]: osdmap e57: 8 total, 8 up, 8 in 2024-03-21T12:32:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:09 smithi172 ceph-mon[29218]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T12:32:09.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:09 smithi172 ceph-mon[24813]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T12:32:09.679 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:32:09.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:09 smithi177 ceph-mon[25767]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T12:32:10.358 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1711024325,"num_in_osds":8,"osd_in_since":1711024308,"num_remapped_pgs":0} 2024-03-21T12:32:10.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T12:32:10.576 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:10 smithi172 ceph-mon[24813]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:10 smithi172 ceph-mon[24813]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T12:32:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:10 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1920052592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:32:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:10 smithi172 ceph-mon[29218]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:10 smithi172 ceph-mon[29218]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T12:32:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:10 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1920052592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:32:10.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:10 smithi177 ceph-mon[25767]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:10 smithi177 ceph-mon[25767]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T12:32:10.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:10 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1920052592' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T12:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:32:12.195 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":59,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","created":"2024-03-21T12:27:20.089381+0000","modified":"2024-03-21T12:32:09.437671+0000","last_up_change":"2024-03-21T12:32:05.965799+0000","last_in_change":"2024-03-21T12:31:48.639333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:30:11.708185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6803","nonce":1201458413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6805","nonce":1201458413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6809","nonce":1201458413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6807","nonce":1201458413}]},"public_addr":"172.21.15.172:6803/1201458413","cluster_addr":"172.21.15.172:6805/1201458413","heartbeat_back_addr":"172.21.15.172:6809/1201458413","heartbeat_front_addr":"172.21.15.172:6807/1201458413","state":["exists","up"]},{"osd":1,"uuid":"8b12aac0-ae0c-48a7-962f-8be8499212dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6811","nonce":2577289746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6813","nonce":2577289746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6817","nonce":2577289746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6815","nonce":2577289746}]},"public_addr":"172.21.15.172:6811/2577289746","cluster_addr":"172.21.15.172:6813/2577289746","heartbeat_back_addr":"172.21.15.172:6817/2577289746","heartbeat_front_addr":"172.21.15.172:6815/2577289746","state":["exists","up"]},{"osd":2,"uuid":"b4530bcd-a09e-4a0d-9903-209bbf2cc166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6819","nonce":387439223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6821","nonce":387439223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6825","nonce":387439223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6823","nonce":387439223}]},"public_addr":"172.21.15.172:6819/387439223","cluster_addr":"172.21.15.172:6821/387439223","heartbeat_back_addr":"172.21.15.172:6825/387439223","heartbeat_front_addr":"172.21.15.172:6823/387439223","state":["exists","up"]},{"osd":3,"uuid":"781e720a-b7ed-47b2-9d17-5b2458d9d5f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6827","nonce":3588711542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6829","nonce":3588711542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6833","nonce":3588711542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6831","nonce":3588711542}]},"public_addr":"172.21.15.172:6827/3588711542","cluster_addr":"172.21.15.172:6829/3588711542","heartbeat_back_addr":"172.21.15.172:6833/3588711542","heartbeat_front_addr":"172.21.15.172:6831/3588711542","state":["exists","up"]},{"osd":4,"uuid":"d5995e0a-38a0-43ba-b9a5-981b861a32d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6801","nonce":3207213564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6803","nonce":3207213564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6807","nonce":3207213564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6805","nonce":3207213564}]},"public_addr":"172.21.15.177:6801/3207213564","cluster_addr":"172.21.15.177:6803/3207213564","heartbeat_back_addr":"172.21.15.177:6807/3207213564","heartbeat_front_addr":"172.21.15.177:6805/3207213564","state":["exists","up"]},{"osd":5,"uuid":"b82613bd-8471-4213-8c7d-e5a9ecde1844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6809","nonce":890621767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6811","nonce":890621767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6815","nonce":890621767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6813","nonce":890621767}]},"public_addr":"172.21.15.177:6809/890621767","cluster_addr":"172.21.15.177:6811/890621767","heartbeat_back_addr":"172.21.15.177:6815/890621767","heartbeat_front_addr":"172.21.15.177:6813/890621767","state":["exists","up"]},{"osd":6,"uuid":"761832a5-706c-4850-b314-f7a0419c729b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6817","nonce":3923649831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6819","nonce":3923649831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6823","nonce":3923649831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6821","nonce":3923649831}]},"public_addr":"172.21.15.177:6817/3923649831","cluster_addr":"172.21.15.177:6819/3923649831","heartbeat_back_addr":"172.21.15.177:6823/3923649831","heartbeat_front_addr":"172.21.15.177:6821/3923649831","state":["exists","up"]},{"osd":7,"uuid":"dc315a72-89dd-4a44-8b88-efbe5562d78b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6825","nonce":3906799781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6827","nonce":3906799781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6831","nonce":3906799781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6829","nonce":3906799781}]},"public_addr":"172.21.15.177:6825/3906799781","cluster_addr":"172.21.15.177:6827/3906799781","heartbeat_back_addr":"172.21.15.177:6831/3906799781","heartbeat_front_addr":"172.21.15.177:6829/3906799781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:29:22.061666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:29:45.325110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:08.885438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:32.429628+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:53.551069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:31:16.531179+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:31:39.966949+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:32:04.061083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:6801/1178568884":"2024-03-22T12:28:05.338868+0000","172.21.15.172:6800/1178568884":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/1643456920":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3553472034":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3623707692":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/422523718":"2024-03-22T12:27:44.194882+0000","172.21.15.172:6800/3189318036":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/1465987532":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3168760134":"2024-03-22T12:27:44.194882+0000","172.21.15.172:6801/3189318036":"2024-03-22T12:27:44.194882+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T12:32:12.507 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:12 smithi172 ceph-mon[24813]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T12:32:12.507 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:12 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1065228124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:12.507 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:12 smithi172 ceph-mon[29218]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T12:32:12.507 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:12 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1065228124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:12.763 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T12:30:11.708185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-21T12:32:12.763 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-21T12:32:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:12 smithi177 ceph-mon[25767]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-03-21T12:32:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:12 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1065228124' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:12.978 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:14.636 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2024-03-21T12:32:14.636 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:14 smithi172 ceph-mon[24813]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:32:14.637 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:14 smithi172 ceph-mon[29218]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:32:14.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:14 smithi177 ceph-mon[25767]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-21T12:32:15.235 INFO:tasks.cephadm:Adding prometheus.a on smithi177 2024-03-21T12:32:15.235 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi177=a' 2024-03-21T12:32:15.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:15 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2955617143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T12:32:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:15 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2955617143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T12:32:15.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:15 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2955617143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T12:32:16.690 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:16 smithi177 ceph-mon[25767]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:32:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:16 smithi172 ceph-mon[24813]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:32:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:16 smithi172 ceph-mon[29218]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-21T12:32:17.089 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled prometheus update... 2024-03-21T12:32:17.764 DEBUG:teuthology.orchestra.run.smithi177:prometheus.a> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-21T12:32:17.766 INFO:tasks.cephadm:Adding node-exporter.a on smithi172 2024-03-21T12:32:17.766 INFO:tasks.cephadm:Adding node-exporter.b on smithi177 2024-03-21T12:32:17.767 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi172=a;smithi177=b' 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ignoring --setuser ceph since I am not root 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ignoring --setgroup ceph since I am not root 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mgr[26950]: -- 172.21.15.177:0/2935806674 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56168acd6340 con 0x56168acaf000 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: Saving service prometheus spec with placement smithi177=a;count:1 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:18.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:18.340 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:18.340 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:18.340 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T12:32:18.340 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-mon[25767]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-21T12:32:18.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setuser ceph since I am not root 2024-03-21T12:32:18.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setgroup ceph since I am not root 2024-03-21T12:32:18.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mgr[25051]: -- 172.21.15.172:0/3816841458 <== mon.1 v2:172.21.15.172:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555a80daa340 con 0x555a80d82800 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: Saving service prometheus spec with placement smithi177=a;count:1 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[24813]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-21T12:32:18.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: Saving service prometheus spec with placement smithi177=a;count:1 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T12:32:18.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-mon[29218]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-03-21T12:32:18.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:18.608+0000 7f29be3e2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:32:18.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:18 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:18.696+0000 7f29be3e2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:32:19.079 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:18.620+0000 7effddb90000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:32:19.079 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:18.710+0000 7effddb90000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:32:19.338 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.316+0000 7effddb90000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:32:19.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-mon[25767]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T12:32:19.339 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-mon[25767]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T12:32:19.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-mon[24813]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T12:32:19.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-mon[24813]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T12:32:19.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-mon[29218]: from='mgr.14150 172.21.15.172:0/557922414' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T12:32:19.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-mon[29218]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T12:32:19.370 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.282+0000 7f29be3e2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:32:19.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.410+0000 7effddb90000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:32:19.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.497+0000 7effddb90000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:32:19.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.369+0000 7f29be3e2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:32:19.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.452+0000 7f29be3e2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:32:19.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.602+0000 7f29be3e2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:32:19.932 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.655+0000 7effddb90000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:32:19.932 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.738+0000 7effddb90000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:32:19.932 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.848+0000 7effddb90000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:32:19.932 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:19.932+0000 7effddb90000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:32:20.032 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.682+0000 7f29be3e2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:32:20.032 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.788+0000 7f29be3e2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:32:20.032 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:19.871+0000 7f29be3e2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:32:20.032 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:20.032+0000 7f29be3e2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:32:20.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:20.094+0000 7effddb90000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:32:20.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:20.179+0000 7effddb90000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:32:20.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:20.120+0000 7f29be3e2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:32:21.522 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:21.201+0000 7effddb90000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:32:21.522 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:21.330+0000 7effddb90000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:32:21.522 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:21.521+0000 7effddb90000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:32:21.524 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:21.187+0000 7f29be3e2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:32:21.524 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:21.332+0000 7f29be3e2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:32:21.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:21.609+0000 7effddb90000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:32:21.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:21.744+0000 7effddb90000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:32:21.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:21.524+0000 7f29be3e2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:32:21.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:21.612+0000 7f29be3e2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:32:21.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:21.748+0000 7f29be3e2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:32:22.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:22 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:22.065+0000 7effddb90000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:32:22.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:22 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:22.283+0000 7effddb90000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:32:22.374 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:22.068+0000 7f29be3e2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:32:22.374 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:22.289+0000 7f29be3e2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:32:22.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:22.374+0000 7f29be3e2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:32:22.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:22.543+0000 7f29be3e2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:32:22.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:22 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:22.367+0000 7effddb90000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:32:22.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:22 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:22.535+0000 7effddb90000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:32:23.741 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:23.448+0000 7effddb90000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:32:23.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:23.466+0000 7f29be3e2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:32:23.999 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:23.741+0000 7effddb90000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:32:23.999 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:23.835+0000 7effddb90000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:32:24.000 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:23.984+0000 7effddb90000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:32:24.000 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: [21/Mar/2024:12:32:24] ENGINE Bus STARTING 2024-03-21T12:32:24.056 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:23.759+0000 7f29be3e2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:32:24.056 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:23.854+0000 7f29be3e2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:32:24.056 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:24.002+0000 7f29be3e2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:32:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: Standby manager daemon x restarted 2024-03-21T12:32:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:32:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:32:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:32:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:32:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: Active manager daemon y restarted 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: Activating manager daemon y 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-mon[25767]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: CherryPy Checker: 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: The Application mounted at '' has an empty config. 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: [21/Mar/2024:12:32:24] ENGINE Serving on http://:::9283 2024-03-21T12:32:24.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: [21/Mar/2024:12:32:24] ENGINE Bus STARTED 2024-03-21T12:32:24.394 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:32:24] ENGINE Bus STARTING 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: Standby manager daemon x restarted 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: Active manager daemon y restarted 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: Activating manager daemon y 2024-03-21T12:32:24.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[24813]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: Standby manager daemon x restarted 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3217876115' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: Active manager daemon y restarted 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: Activating manager daemon y 2024-03-21T12:32:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-mon[29218]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T12:32:24.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: CherryPy Checker: 2024-03-21T12:32:24.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: The Application mounted at '' has an empty config. 2024-03-21T12:32:24.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:32:24.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:32:24] ENGINE Serving on http://:::9283 2024-03-21T12:32:24.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:32:24] ENGINE Bus STARTED 2024-03-21T12:32:25.078 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled node-exporter update... 2024-03-21T12:32:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: mgrmap e17: y(active, starting, since 0.0388178s), standbys: x 2024-03-21T12:32:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:32:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:32:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:32:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: Manager daemon y is now available 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: mgrmap e17: y(active, starting, since 0.0388178s), standbys: x 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:32:25.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:25.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: mgrmap e17: y(active, starting, since 0.0388178s), standbys: x 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:32:25.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: Manager daemon y is now available 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:32:25.368 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:25.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:32:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:32:25] ENGINE Bus STARTING 2024-03-21T12:32:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:32:25] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:32:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:32:25] ENGINE Bus STARTED 2024-03-21T12:32:25.932 DEBUG:teuthology.orchestra.run.smithi172:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-21T12:32:25.935 DEBUG:teuthology.orchestra.run.smithi177:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-21T12:32:25.937 INFO:tasks.cephadm:Adding alertmanager.a on smithi172 2024-03-21T12:32:25.937 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi172=a' 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[24813]: mgrmap e18: y(active, since 1.04817s), standbys: x 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[24813]: Saving service node-exporter spec with placement smithi172=a;smithi177=b;count:2 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[24813]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[29218]: mgrmap e18: y(active, since 1.04817s), standbys: x 2024-03-21T12:32:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[29218]: Saving service node-exporter spec with placement smithi172=a;smithi177=b;count:2 2024-03-21T12:32:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[29218]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:26 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:26 smithi177 ceph-mon[25767]: mgrmap e18: y(active, since 1.04817s), standbys: x 2024-03-21T12:32:26.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:26 smithi177 ceph-mon[25767]: Saving service node-exporter spec with placement smithi172=a;smithi177=b;count:2 2024-03-21T12:32:26.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:26 smithi177 ceph-mon[25767]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:26.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:26 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:26 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:26.294 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:26 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.074 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:32:25] ENGINE Bus STARTING 2024-03-21T12:32:27.326 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:32:25] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:32:27.326 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:32:25] ENGINE Bus STARTED 2024-03-21T12:32:27.326 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:27.327 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.327 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-21T12:32:27.327 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:32:25] ENGINE Bus STARTING 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:32:25] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:32:25] ENGINE Bus STARTED 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-21T12:32:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:32:25] ENGINE Bus STARTING 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:32:25] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:32:25] ENGINE Bus STARTED 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-21T12:32:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:27.986 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled alertmanager update... 2024-03-21T12:32:28.317 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:28 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:32:28.317 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:28 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:32:28.317 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:28 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:28.317 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:28 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:32:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:32:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:28.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:28.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:32:28.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:32:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:28 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:28.623 DEBUG:teuthology.orchestra.run.smithi172:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-21T12:32:28.626 INFO:tasks.cephadm:Adding grafana.a on smithi177 2024-03-21T12:32:28.626 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi177=a' 2024-03-21T12:32:29.115 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:29 smithi177 ceph-mon[25767]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:29.115 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:29 smithi177 ceph-mon[25767]: Saving service alertmanager spec with placement smithi172=a;count:1 2024-03-21T12:32:29.115 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:29 smithi177 ceph-mon[25767]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:29.115 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:29 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[24813]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[24813]: Saving service alertmanager spec with placement smithi172=a;count:1 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[24813]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[29218]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[29218]: Saving service alertmanager spec with placement smithi172=a;count:1 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[29218]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:29 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:30 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:32:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:30 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:32:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:30 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:30 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:32:30.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:32:30.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:32:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:32:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:30 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:30.778 INFO:teuthology.orchestra.run.smithi177.stdout:Scheduled grafana update... 2024-03-21T12:32:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:31 smithi177 ceph-mon[25767]: Deploying daemon prometheus.a on smithi177 2024-03-21T12:32:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:31 smithi177 ceph-mon[25767]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:31 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:31 smithi172 ceph-mon[24813]: Deploying daemon prometheus.a on smithi177 2024-03-21T12:32:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:31 smithi172 ceph-mon[24813]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:31 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:31 smithi172 ceph-mon[29218]: Deploying daemon prometheus.a on smithi177 2024-03-21T12:32:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:31 smithi172 ceph-mon[29218]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:31 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:31.423 DEBUG:teuthology.orchestra.run.smithi177:grafana.a> sudo journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service 2024-03-21T12:32:31.425 INFO:tasks.cephadm:Setting up client nodes... 2024-03-21T12:32:31.425 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T12:32:32.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:32 smithi177 ceph-mon[25767]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:32 smithi177 ceph-mon[25767]: Saving service grafana spec with placement smithi177=a;count:1 2024-03-21T12:32:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:32 smithi172 ceph-mon[24813]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:32 smithi172 ceph-mon[24813]: Saving service grafana spec with placement smithi177=a;count:1 2024-03-21T12:32:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:32 smithi172 ceph-mon[29218]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi177=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:32:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:32 smithi172 ceph-mon[29218]: Saving service grafana spec with placement smithi177=a;count:1 2024-03-21T12:32:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:33 smithi177 ceph-mon[25767]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:33 smithi172 ceph-mon[24813]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:33 smithi172 ceph-mon[29218]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:33.563 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-03-21T12:32:33.563 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQDhKPxlPshSIRAAhAS1zEOYxYp+Ld6waXz84w== 2024-03-21T12:32:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:34 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3963199178' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:34 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:34.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:34 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:32:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3963199178' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:34 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:34 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:32:34.222 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-21T12:32:34.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-21T12:32:34.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T12:32:34.265 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T12:32:34.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:34 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3963199178' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:34 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:34 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:32:35.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:35 smithi177 ceph-mon[25767]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:35.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:35 smithi172 ceph-mon[24813]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:35 smithi172 ceph-mon[29218]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:37.344 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:37 smithi177 ceph-mon[25767]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:37 smithi172 ceph-mon[24813]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:37 smithi172 ceph-mon[29218]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:38.139 INFO:teuthology.orchestra.run.smithi177.stdout:[client.1] 2024-03-21T12:32:38.139 INFO:teuthology.orchestra.run.smithi177.stdout: key = AQDmKPxl+e/8BxAAs7pxJHwRxf9hCWCyR94EHg== 2024-03-21T12:32:39.138 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:39 smithi177 ceph-mon[25767]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:39.138 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.177:0/3643422258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:39.138 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:39.138 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[24813]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.177:0/3643422258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[29218]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.177:0/3643422258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T12:32:39.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T12:32:39.453 DEBUG:teuthology.orchestra.run.smithi177:> set -ex 2024-03-21T12:32:39.453 DEBUG:teuthology.orchestra.run.smithi177:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-21T12:32:39.453 DEBUG:teuthology.orchestra.run.smithi177:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-21T12:32:39.496 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T12:32:39.496 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-21T12:32:39.496 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-21T12:32:39.720 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:41.094 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:41 smithi172 ceph-mon[24813]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:41.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:41 smithi177 ceph-mon[25767]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:41 smithi172 ceph-mon[29218]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:41.551 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:32:41.782 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:41 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:32:42.036 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:41 smithi177 podman[40892]: 2024-03-21 12:32:41.686927508 +0000 UTC m=+0.018562206 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T12:32:42.036 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:41 smithi177 podman[40892]: 2024-03-21 12:32:41.800786812 +0000 UTC m=+0.132421497 container create ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:32:42.036 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 podman[40892]: 2024-03-21 12:32:42.011329208 +0000 UTC m=+0.342963894 container init ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:32:42.036 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 podman[40892]: 2024-03-21 12:32:42.013855293 +0000 UTC m=+0.345489970 container start ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:32:42.168 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":19,"active_gid":14421,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3184130888},{"type":"v1","addr":"172.21.15.172:6801","nonce":3184130888}]},"active_addr":"172.21.15.172:6801/3184130888","active_change":"2024-03-21T12:32:24.015909+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34319,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.172:8443/","prometheus":"http://172.21.15.172:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":623202746}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":855901005}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":950482550}]},{"addrvec":[{"type":"v2","addr":"172.21.15.172:0","nonce":1132277964}]}]}} 2024-03-21T12:32:42.171 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-21T12:32:42.171 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T12:32:42.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T12:32:42.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.036Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T12:32:42.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.037Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T12:32:42.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.037Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T12:32:42.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.037Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:32:42.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.037Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.037Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.038Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.038Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.039Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.041Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.041Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.61µs 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.041Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.041Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.041Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.724µs wal_replay_duration=118.536µs total_replay_duration=150.953µs 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.042Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.042Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.042Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.058Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.364469ms db_storage=570ns remote_storage=1.266µs web_handler=232ns query_engine=588ns scrape=408.549µs scrape_sd=14.551µs notify=875ns notify_sd=1.329µs rules=15.777503ms 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:32:42.058Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 bash[40892]: ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a 2024-03-21T12:32:42.359 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:32:42 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:32:42.413 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:42.675 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:42 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/815614062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:32:42.675 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:42 smithi172 ceph-mon[24813]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:42.675 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:42 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:42.676 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:42 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/815614062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:32:42.676 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:42 smithi172 ceph-mon[29218]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:42.676 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:42 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/815614062' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:32:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-mon[25767]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:42 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:43.688 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:43 smithi172 ceph-mon[24813]: Deploying daemon node-exporter.a on smithi172 2024-03-21T12:32:43.688 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:43 smithi172 ceph-mon[29218]: Deploying daemon node-exporter.a on smithi172 2024-03-21T12:32:43.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:43 smithi177 ceph-mon[25767]: Deploying daemon node-exporter.a on smithi172 2024-03-21T12:32:44.214 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:32:44.214 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":60,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","created":"2024-03-21T12:27:20.089381+0000","modified":"2024-03-21T12:32:24.015539+0000","last_up_change":"2024-03-21T12:32:05.965799+0000","last_in_change":"2024-03-21T12:31:48.639333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:30:11.708185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6803","nonce":1201458413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6805","nonce":1201458413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6809","nonce":1201458413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6807","nonce":1201458413}]},"public_addr":"172.21.15.172:6803/1201458413","cluster_addr":"172.21.15.172:6805/1201458413","heartbeat_back_addr":"172.21.15.172:6809/1201458413","heartbeat_front_addr":"172.21.15.172:6807/1201458413","state":["exists","up"]},{"osd":1,"uuid":"8b12aac0-ae0c-48a7-962f-8be8499212dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6811","nonce":2577289746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6813","nonce":2577289746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6817","nonce":2577289746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6815","nonce":2577289746}]},"public_addr":"172.21.15.172:6811/2577289746","cluster_addr":"172.21.15.172:6813/2577289746","heartbeat_back_addr":"172.21.15.172:6817/2577289746","heartbeat_front_addr":"172.21.15.172:6815/2577289746","state":["exists","up"]},{"osd":2,"uuid":"b4530bcd-a09e-4a0d-9903-209bbf2cc166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6819","nonce":387439223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6821","nonce":387439223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6825","nonce":387439223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6823","nonce":387439223}]},"public_addr":"172.21.15.172:6819/387439223","cluster_addr":"172.21.15.172:6821/387439223","heartbeat_back_addr":"172.21.15.172:6825/387439223","heartbeat_front_addr":"172.21.15.172:6823/387439223","state":["exists","up"]},{"osd":3,"uuid":"781e720a-b7ed-47b2-9d17-5b2458d9d5f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6827","nonce":3588711542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6829","nonce":3588711542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6833","nonce":3588711542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6831","nonce":3588711542}]},"public_addr":"172.21.15.172:6827/3588711542","cluster_addr":"172.21.15.172:6829/3588711542","heartbeat_back_addr":"172.21.15.172:6833/3588711542","heartbeat_front_addr":"172.21.15.172:6831/3588711542","state":["exists","up"]},{"osd":4,"uuid":"d5995e0a-38a0-43ba-b9a5-981b861a32d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6801","nonce":3207213564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6803","nonce":3207213564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6807","nonce":3207213564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6805","nonce":3207213564}]},"public_addr":"172.21.15.177:6801/3207213564","cluster_addr":"172.21.15.177:6803/3207213564","heartbeat_back_addr":"172.21.15.177:6807/3207213564","heartbeat_front_addr":"172.21.15.177:6805/3207213564","state":["exists","up"]},{"osd":5,"uuid":"b82613bd-8471-4213-8c7d-e5a9ecde1844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6809","nonce":890621767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6811","nonce":890621767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6815","nonce":890621767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6813","nonce":890621767}]},"public_addr":"172.21.15.177:6809/890621767","cluster_addr":"172.21.15.177:6811/890621767","heartbeat_back_addr":"172.21.15.177:6815/890621767","heartbeat_front_addr":"172.21.15.177:6813/890621767","state":["exists","up"]},{"osd":6,"uuid":"761832a5-706c-4850-b314-f7a0419c729b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6817","nonce":3923649831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6819","nonce":3923649831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6823","nonce":3923649831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6821","nonce":3923649831}]},"public_addr":"172.21.15.177:6817/3923649831","cluster_addr":"172.21.15.177:6819/3923649831","heartbeat_back_addr":"172.21.15.177:6823/3923649831","heartbeat_front_addr":"172.21.15.177:6821/3923649831","state":["exists","up"]},{"osd":7,"uuid":"dc315a72-89dd-4a44-8b88-efbe5562d78b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6825","nonce":3906799781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6827","nonce":3906799781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6831","nonce":3906799781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6829","nonce":3906799781}]},"public_addr":"172.21.15.177:6825/3906799781","cluster_addr":"172.21.15.177:6827/3906799781","heartbeat_back_addr":"172.21.15.177:6831/3906799781","heartbeat_front_addr":"172.21.15.177:6829/3906799781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:29:22.061666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:29:45.325110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:08.885438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:32.429628+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:53.551069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:31:16.531179+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:31:39.966949+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:32:04.061083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:0/4048413631":"2024-03-22T12:32:24.015501+0000","172.21.15.172:0/238925532":"2024-03-22T12:32:24.015501+0000","172.21.15.172:6801/3682709254":"2024-03-22T12:32:24.015501+0000","172.21.15.172:6801/1178568884":"2024-03-22T12:28:05.338868+0000","172.21.15.172:6800/1178568884":"2024-03-22T12:28:05.338868+0000","172.21.15.172:6800/3682709254":"2024-03-22T12:32:24.015501+0000","172.21.15.172:0/1643456920":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3553472034":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/4216383095":"2024-03-22T12:32:24.015501+0000","172.21.15.172:0/3623707692":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/422523718":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/4198175409":"2024-03-22T12:32:24.015501+0000","172.21.15.172:6800/3189318036":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/1465987532":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3168760134":"2024-03-22T12:27:44.194882+0000","172.21.15.172:6801/3189318036":"2024-03-22T12:27:44.194882+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T12:32:44.860 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:44 smithi172 bash[44226]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-21T12:32:44.885 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T12:32:44.886 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T12:32:45.138 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:45 smithi172 ceph-mon[24813]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:45 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:45 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/544368503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:45 smithi172 ceph-mon[29218]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:45 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:45 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/544368503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:45.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:45 smithi177 ceph-mon[25767]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:45.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:45 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:45.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:45 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/544368503' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:46.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:46 smithi172 bash[44226]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-21T12:32:47.085 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:46 smithi172 bash[44226]: Writing manifest to image destination 2024-03-21T12:32:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-mon[24813]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-mon[29218]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:47.396 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 podman[44226]: 2024-03-21 12:32:47.112114632 +0000 UTC m=+3.285276472 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T12:32:47.396 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 podman[44226]: 2024-03-21 12:32:47.227113496 +0000 UTC m=+3.400275327 container create 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:32:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:47 smithi177 ceph-mon[25767]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 podman[44226]: 2024-03-21 12:32:47.429555284 +0000 UTC m=+3.602717116 container init 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 podman[44226]: 2024-03-21 12:32:47.432972131 +0000 UTC m=+3.606133966 container start 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.437Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.437Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T12:32:47.680 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T12:32:47.681 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T12:32:47.682 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T12:32:47.683 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T12:32:47.684 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.438Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T12:32:47.684 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[44415]: ts=2024-03-21T12:32:47.439Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T12:32:47.684 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 bash[44226]: 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc 2024-03-21T12:32:47.684 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:32:47 smithi172 systemd[1]: Started Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:32:48.236 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:32:48.236 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":60,"fsid":"3bbfe794-e77e-11ee-95cd-87774f69a715","created":"2024-03-21T12:27:20.089381+0000","modified":"2024-03-21T12:32:24.015539+0000","last_up_change":"2024-03-21T12:32:05.965799+0000","last_in_change":"2024-03-21T12:31:48.639333+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T12:30:11.708185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fdbc221c-a7f3-4e6d-a726-d83d7a4e73b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6803","nonce":1201458413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6805","nonce":1201458413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6809","nonce":1201458413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":1201458413},{"type":"v1","addr":"172.21.15.172:6807","nonce":1201458413}]},"public_addr":"172.21.15.172:6803/1201458413","cluster_addr":"172.21.15.172:6805/1201458413","heartbeat_back_addr":"172.21.15.172:6809/1201458413","heartbeat_front_addr":"172.21.15.172:6807/1201458413","state":["exists","up"]},{"osd":1,"uuid":"8b12aac0-ae0c-48a7-962f-8be8499212dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6811","nonce":2577289746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6813","nonce":2577289746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6817","nonce":2577289746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2577289746},{"type":"v1","addr":"172.21.15.172:6815","nonce":2577289746}]},"public_addr":"172.21.15.172:6811/2577289746","cluster_addr":"172.21.15.172:6813/2577289746","heartbeat_back_addr":"172.21.15.172:6817/2577289746","heartbeat_front_addr":"172.21.15.172:6815/2577289746","state":["exists","up"]},{"osd":2,"uuid":"b4530bcd-a09e-4a0d-9903-209bbf2cc166","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6819","nonce":387439223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6821","nonce":387439223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6825","nonce":387439223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":387439223},{"type":"v1","addr":"172.21.15.172:6823","nonce":387439223}]},"public_addr":"172.21.15.172:6819/387439223","cluster_addr":"172.21.15.172:6821/387439223","heartbeat_back_addr":"172.21.15.172:6825/387439223","heartbeat_front_addr":"172.21.15.172:6823/387439223","state":["exists","up"]},{"osd":3,"uuid":"781e720a-b7ed-47b2-9d17-5b2458d9d5f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6827","nonce":3588711542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6829","nonce":3588711542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6833","nonce":3588711542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3588711542},{"type":"v1","addr":"172.21.15.172:6831","nonce":3588711542}]},"public_addr":"172.21.15.172:6827/3588711542","cluster_addr":"172.21.15.172:6829/3588711542","heartbeat_back_addr":"172.21.15.172:6833/3588711542","heartbeat_front_addr":"172.21.15.172:6831/3588711542","state":["exists","up"]},{"osd":4,"uuid":"d5995e0a-38a0-43ba-b9a5-981b861a32d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6800","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6801","nonce":3207213564}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6802","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6803","nonce":3207213564}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6806","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6807","nonce":3207213564}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6804","nonce":3207213564},{"type":"v1","addr":"172.21.15.177:6805","nonce":3207213564}]},"public_addr":"172.21.15.177:6801/3207213564","cluster_addr":"172.21.15.177:6803/3207213564","heartbeat_back_addr":"172.21.15.177:6807/3207213564","heartbeat_front_addr":"172.21.15.177:6805/3207213564","state":["exists","up"]},{"osd":5,"uuid":"b82613bd-8471-4213-8c7d-e5a9ecde1844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6808","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6809","nonce":890621767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6810","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6811","nonce":890621767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6814","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6815","nonce":890621767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6812","nonce":890621767},{"type":"v1","addr":"172.21.15.177:6813","nonce":890621767}]},"public_addr":"172.21.15.177:6809/890621767","cluster_addr":"172.21.15.177:6811/890621767","heartbeat_back_addr":"172.21.15.177:6815/890621767","heartbeat_front_addr":"172.21.15.177:6813/890621767","state":["exists","up"]},{"osd":6,"uuid":"761832a5-706c-4850-b314-f7a0419c729b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6816","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6817","nonce":3923649831}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6818","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6819","nonce":3923649831}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6822","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6823","nonce":3923649831}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6820","nonce":3923649831},{"type":"v1","addr":"172.21.15.177:6821","nonce":3923649831}]},"public_addr":"172.21.15.177:6817/3923649831","cluster_addr":"172.21.15.177:6819/3923649831","heartbeat_back_addr":"172.21.15.177:6823/3923649831","heartbeat_front_addr":"172.21.15.177:6821/3923649831","state":["exists","up"]},{"osd":7,"uuid":"dc315a72-89dd-4a44-8b88-efbe5562d78b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6824","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6825","nonce":3906799781}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6826","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6827","nonce":3906799781}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6830","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6831","nonce":3906799781}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.177:6828","nonce":3906799781},{"type":"v1","addr":"172.21.15.177:6829","nonce":3906799781}]},"public_addr":"172.21.15.177:6825/3906799781","cluster_addr":"172.21.15.177:6827/3906799781","heartbeat_back_addr":"172.21.15.177:6831/3906799781","heartbeat_front_addr":"172.21.15.177:6829/3906799781","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:29:22.061666+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:29:45.325110+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:08.885438+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:32.429628+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:30:53.551069+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:31:16.531179+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:31:39.966949+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T12:32:04.061083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:0/4048413631":"2024-03-22T12:32:24.015501+0000","172.21.15.172:0/238925532":"2024-03-22T12:32:24.015501+0000","172.21.15.172:6801/3682709254":"2024-03-22T12:32:24.015501+0000","172.21.15.172:6801/1178568884":"2024-03-22T12:28:05.338868+0000","172.21.15.172:6800/1178568884":"2024-03-22T12:28:05.338868+0000","172.21.15.172:6800/3682709254":"2024-03-22T12:32:24.015501+0000","172.21.15.172:0/1643456920":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3553472034":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/4216383095":"2024-03-22T12:32:24.015501+0000","172.21.15.172:0/3623707692":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/422523718":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/4198175409":"2024-03-22T12:32:24.015501+0000","172.21.15.172:6800/3189318036":"2024-03-22T12:27:44.194882+0000","172.21.15.172:0/1465987532":"2024-03-22T12:28:05.338868+0000","172.21.15.172:0/3168760134":"2024-03-22T12:27:44.194882+0000","172.21.15.172:6801/3189318036":"2024-03-22T12:27:44.194882+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T12:32:48.607 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:48 smithi177 bash[41143]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-21T12:32:48.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:48 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:48 smithi177 ceph-mon[25767]: Deploying daemon node-exporter.b on smithi177 2024-03-21T12:32:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:48 smithi177 ceph-mon[25767]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:48 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/94326702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[24813]: Deploying daemon node-exporter.b on smithi177 2024-03-21T12:32:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[24813]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/94326702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[29218]: Deploying daemon node-exporter.b on smithi177 2024-03-21T12:32:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[29218]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:48 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/94326702' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T12:32:48.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T12:32:48.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T12:32:48.978 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T12:32:48.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T12:32:48.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T12:32:48.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T12:32:48.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T12:32:48.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T12:32:49.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:48 smithi177 bash[41143]: Getting image source signatures 2024-03-21T12:32:49.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:48 smithi177 bash[41143]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-21T12:32:49.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:48 smithi177 bash[41143]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-21T12:32:49.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:48 smithi177 bash[41143]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-21T12:32:49.363 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.394 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.395 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.400 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.576 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.579 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.580 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:49.580 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:32:50.857 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:50 smithi177 bash[41143]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-21T12:32:50.858 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:50 smithi177 bash[41143]: Writing manifest to image destination 2024-03-21T12:32:51.116 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:32:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:32:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:32:51.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:51 smithi172 ceph-mon[24813]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:51 smithi172 ceph-mon[29218]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:51.435 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-mon[25767]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:51.435 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 podman[41143]: 2024-03-21 12:32:51.216152631 +0000 UTC m=+2.839568306 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T12:32:51.435 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 podman[41143]: 2024-03-21 12:32:51.306310135 +0000 UTC m=+2.929725803 container create 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:32:51.858 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 podman[41143]: 2024-03-21 12:32:51.525245849 +0000 UTC m=+3.148661527 container init 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 podman[41143]: 2024-03-21 12:32:51.528688025 +0000 UTC m=+3.152103693 container start 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.532Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.532Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T12:32:51.859 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T12:32:51.860 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.533Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T12:32:51.860 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T12:32:51.860 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T12:32:51.860 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T12:32:51.860 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T12:32:51.860 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T12:32:51.861 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T12:32:51.862 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[41194]: ts=2024-03-21T12:32:51.534Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 bash[41143]: 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 2024-03-21T12:32:51.863 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:32:51 smithi177 systemd[1]: Started Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:32:52.955 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:52 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:52.955 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:52 smithi172 ceph-mon[24813]: Deploying daemon alertmanager.a on smithi172 2024-03-21T12:32:52.955 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:52 smithi172 ceph-mon[24813]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:52.956 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:52 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:52.956 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:52 smithi172 ceph-mon[29218]: Deploying daemon alertmanager.a on smithi172 2024-03-21T12:32:52.956 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:52 smithi172 ceph-mon[29218]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:52 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:52 smithi177 ceph-mon[25767]: Deploying daemon alertmanager.a on smithi172 2024-03-21T12:32:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:52 smithi177 ceph-mon[25767]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:55.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:55 smithi172 ceph-mon[24813]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:55.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:55 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:55 smithi172 ceph-mon[29218]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:55 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:55 smithi177 ceph-mon[25767]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:55 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:32:55.944 INFO:teuthology.orchestra.run.smithi172.stdout:55834574888 2024-03-21T12:32:55.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T12:32:57.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:32:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:32:56] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-03-21T12:32:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:57 smithi172 ceph-mon[24813]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:57 smithi172 ceph-mon[29218]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:57.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:57 smithi177 ceph-mon[25767]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:32:59 smithi172 ceph-mon[24813]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:32:59 smithi172 ceph-mon[29218]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:32:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:32:59 smithi177 ceph-mon[25767]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:01.132 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:33:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:33:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:01 smithi172 ceph-mon[29218]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:01 smithi172 ceph-mon[24813]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:01.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:01 smithi177 ceph-mon[25767]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:02 smithi172 ceph-mon[24813]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:02 smithi172 ceph-mon[29218]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:02 smithi177 ceph-mon[25767]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:03.055 INFO:teuthology.orchestra.run.smithi172.stdout:77309411365 2024-03-21T12:33:03.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T12:33:03.980 INFO:teuthology.orchestra.run.smithi172.stdout:34359738412 2024-03-21T12:33:03.980 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T12:33:05.272 INFO:teuthology.orchestra.run.smithi172.stdout:236223201293 2024-03-21T12:33:05.272 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T12:33:05.375 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:05 smithi172 ceph-mon[24813]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:05.375 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:05 smithi172 ceph-mon[29218]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:05.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:05 smithi177 ceph-mon[25767]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:06.593 INFO:teuthology.orchestra.run.smithi172.stdout:133143986203 2024-03-21T12:33:06.593 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T12:33:06.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:33:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:06] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-03-21T12:33:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:07 smithi172 ceph-mon[24813]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:07 smithi172 ceph-mon[29218]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:07.434 INFO:teuthology.orchestra.run.smithi172.stdout:167503724567 2024-03-21T12:33:07.434 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T12:33:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:07 smithi177 ceph-mon[25767]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:07.770 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:08.195 INFO:teuthology.orchestra.run.smithi172.stdout:201863462931 2024-03-21T12:33:08.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T12:33:08.853 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:09.032 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:09.263 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:09 smithi172 ceph-mon[24813]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:09.264 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:09 smithi172 ceph-mon[29218]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:09.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:09 smithi177 ceph-mon[25767]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:09.786 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:09.786 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:10.845 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:10.846 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:11.126 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:33:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:33:11.370 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:11 smithi172 ceph-mon[24813]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:11.371 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:11 smithi172 ceph-mon[29218]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:11.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:11 smithi177 ceph-mon[25767]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:12.267 INFO:teuthology.orchestra.run.smithi172.stdout:111669149729 2024-03-21T12:33:12.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T12:33:12.496 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:12 smithi172 ceph-mon[24813]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:12.496 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:12 smithi172 ceph-mon[29218]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:12.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:12 smithi177 ceph-mon[25767]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:13.471 INFO:teuthology.orchestra.run.smithi172.stdout:77309411367 2024-03-21T12:33:13.810 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:13 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/847625993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T12:33:13.811 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:13 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/847625993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T12:33:13.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:13 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/847625993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T12:33:14.606 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411367 for osd.2 2024-03-21T12:33:14.606 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:14.785 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:14 smithi172 ceph-mon[24813]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:14.786 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:14 smithi172 ceph-mon[29218]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:14.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:14 smithi177 ceph-mon[25767]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:15.574 INFO:teuthology.orchestra.run.smithi172.stdout:55834574892 2024-03-21T12:33:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:15 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3172784242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T12:33:15.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:15 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3172784242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T12:33:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:15 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3172784242' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T12:33:16.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:16 smithi172 ceph-mon[24813]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:16.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:33:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:16] "GET /metrics HTTP/1.1" 200 191500 "" "Prometheus/2.33.4" 2024-03-21T12:33:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:16 smithi172 ceph-mon[29218]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:17.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:16 smithi177 ceph-mon[25767]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:17.956 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574892 for osd.1 2024-03-21T12:33:17.956 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:19 smithi172 ceph-mon[24813]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:19 smithi172 ceph-mon[29218]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:19.520 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:19.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:19 smithi177 ceph-mon[25767]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:21.124 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:33:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:33:21.260 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:21 smithi172 ceph-mon[24813]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:21.260 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:21 smithi172 ceph-mon[29218]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:21.378 INFO:teuthology.orchestra.run.smithi172.stdout:167503724570 2024-03-21T12:33:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:21 smithi177 ceph-mon[25767]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:22.207 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:22 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1813407431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T12:33:22.207 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:22 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1813407431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T12:33:22.412 INFO:teuthology.orchestra.run.smithi172.stdout:34359738416 2024-03-21T12:33:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:22 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1813407431' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T12:33:23.222 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:23 smithi172 ceph-mon[24813]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:23.223 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:23 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/542879768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T12:33:23.223 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:23 smithi172 ceph-mon[29218]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:23.223 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:23 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/542879768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T12:33:23.248 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738412 got 34359738416 for osd.0 2024-03-21T12:33:23.248 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:23.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:23 smithi177 ceph-mon[25767]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:23.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:23 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/542879768' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T12:33:23.845 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[46806]: level=info ts=2024-03-21T12:33:23.790Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-21T12:33:23.845 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:23 smithi172 bash[46725]: 5bee4969042066f3ac966b7d5268bdc972d3bda9f6dffdb5067d3853ab8c8eda 2024-03-21T12:33:24.146 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:23 smithi172 systemd[1]: Started Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:33:25.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:25.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:33:25.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:25.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:25.005 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:33:25.005 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:33:25.005 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:33:25.005 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:33:25.058 INFO:teuthology.orchestra.run.smithi172.stdout:133143986207 2024-03-21T12:33:25.287 INFO:teuthology.orchestra.run.smithi172.stdout:236223201297 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:33:25.337 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:33:25.338 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:33:25.502 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724570 for osd.5 2024-03-21T12:33:25.503 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:25.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[46806]: level=info ts=2024-03-21T12:33:25.762Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000256377s 2024-03-21T12:33:26.116 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986207 for osd.4 2024-03-21T12:33:26.116 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:26.178 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:25 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2937775371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T12:33:26.178 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:25 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1590662959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T12:33:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:25 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2937775371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T12:33:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:25 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1590662959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T12:33:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:25 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2937775371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T12:33:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:25 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1590662959' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T12:33:27.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:26 smithi172 ceph-mon[24813]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:27.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:26 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:27.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:33:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:26] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-03-21T12:33:27.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:26 smithi172 ceph-mon[29218]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:27.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:26 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:27.086 INFO:teuthology.orchestra.run.smithi172.stdout:201863462935 2024-03-21T12:33:27.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:26 smithi177 ceph-mon[25767]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:26 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:27.711 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201293 got 236223201297 for osd.7 2024-03-21T12:33:27.711 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:28.028 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:27 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3694280082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T12:33:28.028 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:27 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3694280082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T12:33:28.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:27 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3694280082' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T12:33:29.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:28 smithi172 ceph-mon[24813]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:29.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:28 smithi172 ceph-mon[29218]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:29.297 INFO:teuthology.orchestra.run.smithi172.stdout:111669149733 2024-03-21T12:33:29.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:28 smithi177 ceph-mon[25767]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:29.543 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462931 got 201863462935 for osd.6 2024-03-21T12:33:29.543 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:30.049 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:29 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/99345706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T12:33:30.050 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:29 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/99345706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T12:33:30.071 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149729 got 111669149733 for osd.3 2024-03-21T12:33:30.071 DEBUG:teuthology.parallel:result is None 2024-03-21T12:33:30.072 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T12:33:30.072 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T12:33:30.303 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:29 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/99345706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T12:33:31.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:30 smithi172 ceph-mon[24813]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:31.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:30 smithi172 ceph-mon[29218]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:31.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:30 smithi177 ceph-mon[25767]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:33:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:33:32.995 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.996 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.997 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:32.997 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:33:32.997 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:32 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:33:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:32 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:33.385 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:33:33.387 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-21T12:33:34.009 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[46806]: level=info ts=2024-03-21T12:33:33.765Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.00311961s 2024-03-21T12:33:34.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:33 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:33:34.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:33 smithi172 ceph-mon[24813]: Deploying daemon grafana.a on smithi177 2024-03-21T12:33:34.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:33 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:33:34.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:33 smithi172 ceph-mon[29218]: Deploying daemon grafana.a on smithi177 2024-03-21T12:33:34.031 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-21T12:33:32.070532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48952,"kb_used_data":6136,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912264,"statfs":{"total":767960285184,"available":767910158336,"internally_reserved":0,"allocated":6283264,"data_stored":3509770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001972"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-21T12:32:24.151634+0000","last_change":"2024-03-21T12:32:09.448629+0000","last_active":"2024-03-21T12:32:24.151634+0000","last_peered":"2024-03-21T12:32:24.151634+0000","last_clean":"2024-03-21T12:32:24.151634+0000","last_became_active":"2024-03-21T12:32:09.448315+0000","last_became_peered":"2024-03-21T12:32:09.448315+0000","last_unstale":"2024-03-21T12:32:24.151634+0000","last_undegraded":"2024-03-21T12:32:24.151634+0000","last_fullsized":"2024-03-21T12:32:24.151634+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:30:12.428218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:30:12.428218+0000","last_clean_scrub_stamp":"2024-03-21T12:30:12.428218+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:53.999175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":1118208,"data_stored":769265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.57199999999999995}]},{"osd":1,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.502}]},{"osd":4,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55300000000000005}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.623},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.501,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.373,"15min":0.36299999999999999},"min":{"1min":0.28000000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.255}]},{"osd":2,"last update":"Thu Mar 21 12:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38300000000000001,"15min":0.38800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.51200000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.312},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.43099999999999999},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51300000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30499999999999999}]},{"osd":3,"last update":"Thu Mar 21 12:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55700000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61899999999999999,"5min":0.624,"15min":0.624},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Mar 21 12:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.66800000000000004,"5min":0.751,"15min":0.751},"last":0.434},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47799999999999998}]},{"osd":5,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.377,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Thu Mar 21 12:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Mar 21 12:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.497},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.45200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 21 12:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.45300000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.498,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.308}]},{"osd":2,"last update":"Thu Mar 21 12:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39800000000000002,"15min":0.40100000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.44}]},{"osd":3,"last update":"Thu Mar 21 12:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.46100000000000002,"15min":0.46400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Mar 21 12:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.40899999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.751,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.65100000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Thu Mar 21 12:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.629}]},{"osd":6,"last update":"Thu Mar 21 12:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Mar 21 12:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.68000000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.35399999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Mar 21 12:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Mar 21 12:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.59099999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Mar 21 12:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Mar 21 12:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.57199999999999995},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Mar 21 12:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38900000000000001,"15min":0.39100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Mar 21 12:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42099999999999999,"15min":0.41899999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.435,"15min":0.437},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.55200000000000005,"5min":0.626,"15min":0.626},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Mar 21 12:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.27500000000000002}]},{"osd":4,"last update":"Thu Mar 21 12:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71199999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Mar 21 12:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.67100000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70999999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Mar 21 12:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56399999999999995}]},{"osd":7,"last update":"Thu Mar 21 12:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.63300000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5764,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739388,"statfs":{"total":95995035648,"available":95989133312,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52400000000000002}]},{"osd":1,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67100000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66200000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.65800000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66400000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64700000000000002,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Mar 21 12:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34999999999999998}]},{"osd":6,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.498},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.371}]},{"osd":7,"last update":"Thu Mar 21 12:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47799999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.57399999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.374},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.61499999999999999,"15min":0.57499999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.6940000000000008,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.36799999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.627,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.61199999999999999,"15min":0.56799999999999995},"min":{"1min":0.39600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":8.6699999999999999,"5min":8.6699999999999999,"15min":8.6699999999999999},"last":1.7250000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70299999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.434}]},{"osd":3,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69499999999999995,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71199999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.622,"5min":0.75,"15min":0.75},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.443},"min":{"1min":0.30099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42099999999999999}]},{"osd":6,"last update":"Thu Mar 21 12:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Mar 21 12:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T12:33:34.033 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T12:33:34.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:33 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:33:34.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:33 smithi177 ceph-mon[25767]: Deploying daemon grafana.a on smithi177 2024-03-21T12:33:34.248 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:35.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:34 smithi177 ceph-mon[25767]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:33:35.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:34 smithi177 ceph-mon[25767]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:34 smithi172 ceph-mon[24813]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:33:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:34 smithi172 ceph-mon[24813]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:34 smithi172 ceph-mon[29218]: from='client.24487 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:33:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:34 smithi172 ceph-mon[29218]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:35.875 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:33:35.877 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-03-21T12:33:36.472 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-21T12:33:34.070844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48952,"kb_used_data":6136,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912264,"statfs":{"total":767960285184,"available":767910158336,"internally_reserved":0,"allocated":6283264,"data_stored":3509770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001985"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":266,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-03-21T12:32:24.151634+0000","last_change":"2024-03-21T12:32:09.448629+0000","last_active":"2024-03-21T12:32:24.151634+0000","last_peered":"2024-03-21T12:32:24.151634+0000","last_clean":"2024-03-21T12:32:24.151634+0000","last_became_active":"2024-03-21T12:32:09.448315+0000","last_became_peered":"2024-03-21T12:32:09.448315+0000","last_unstale":"2024-03-21T12:32:24.151634+0000","last_undegraded":"2024-03-21T12:32:24.151634+0000","last_fullsized":"2024-03-21T12:32:24.151634+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T12:30:12.428218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T12:30:12.428218+0000","last_clean_scrub_stamp":"2024-03-21T12:30:12.428218+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-22T15:06:53.999175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":1118208,"data_stored":769265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.57199999999999995}]},{"osd":1,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.54400000000000004}]},{"osd":2,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56000000000000005}]},{"osd":3,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.502}]},{"osd":4,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42799999999999999}]},{"osd":5,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.47799999999999998}]},{"osd":6,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.55300000000000005}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.623},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.45500000000000002}]},{"osd":5,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.24299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.501,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.373,"15min":0.36299999999999999},"min":{"1min":0.28000000000000003,"5min":0.217,"15min":0.217},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.255}]},{"osd":2,"last update":"Thu Mar 21 12:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.38300000000000001,"15min":0.38800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.51200000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.312},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.42299999999999999,"15min":0.43099999999999999},"min":{"1min":0.28199999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.51300000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.30499999999999999}]},{"osd":3,"last update":"Thu Mar 21 12:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.44700000000000001,"15min":0.45300000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55700000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.48299999999999998,"15min":0.48899999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.61899999999999999,"5min":0.624,"15min":0.624},"last":0.46800000000000003}]},{"osd":4,"last update":"Thu Mar 21 12:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40100000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.66800000000000004,"5min":0.751,"15min":0.751},"last":0.434},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.47799999999999998}]},{"osd":5,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.377,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55100000000000005,"15min":0.54600000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Thu Mar 21 12:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.40899999999999997}]},{"osd":7,"last update":"Thu Mar 21 12:33:07 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.497},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.45200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6924,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738228,"statfs":{"total":95995035648,"available":95987945472,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 21 12:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34399999999999997,"15min":0.34100000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.45300000000000001,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35399999999999998,"15min":0.34699999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.498,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.308}]},{"osd":2,"last update":"Thu Mar 21 12:33:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.39800000000000002,"15min":0.40100000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.38500000000000001,"15min":0.38700000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.44}]},{"osd":3,"last update":"Thu Mar 21 12:32:37 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.46100000000000002,"15min":0.46400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.44700000000000001,"15min":0.45100000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Mar 21 12:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.56499999999999995,"15min":0.56899999999999995},"min":{"1min":0.40899999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.751,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.65100000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.69099999999999995}]},{"osd":5,"last update":"Thu Mar 21 12:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.58699999999999997,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.629}]},{"osd":6,"last update":"Thu Mar 21 12:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.59399999999999997}]},{"osd":7,"last update":"Thu Mar 21 12:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.68000000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35999999999999999,"15min":0.35399999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.44800000000000001}]},{"osd":1,"last update":"Thu Mar 21 12:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38200000000000001,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38,"15min":0.36699999999999999},"min":{"1min":0.28199999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Mar 21 12:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.57299999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.44400000000000001,"15min":0.45000000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.59099999999999997,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.40799999999999997}]},{"osd":4,"last update":"Thu Mar 21 12:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.54700000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73199999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Mar 21 12:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57599999999999996,"15min":0.57199999999999995},"min":{"1min":0.45100000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.435,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.66300000000000003}]},{"osd":6,"last update":"Thu Mar 21 12:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.66800000000000004}]},{"osd":7,"last update":"Thu Mar 21 12:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.64800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38900000000000001,"15min":0.39100000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.51100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Mar 21 12:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42099999999999999,"15min":0.41899999999999998},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.435,"15min":0.437},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.55200000000000005,"5min":0.626,"15min":0.626},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Mar 21 12:32:39 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.46999999999999997,"15min":0.47199999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.27500000000000002}]},{"osd":4,"last update":"Thu Mar 21 12:32:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.56299999999999994,"15min":0.56399999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71199999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66400000000000003}]},{"osd":5,"last update":"Thu Mar 21 12:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58699999999999997,"15min":0.59599999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.67100000000000004,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70999999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Mar 21 12:32:44 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.56399999999999995}]},{"osd":7,"last update":"Thu Mar 21 12:33:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.63300000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5764,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739388,"statfs":{"total":95995035648,"available":95989133312,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.54900000000000004,"15min":0.55900000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55600000000000005,"15min":0.56599999999999995},"min":{"1min":0.36399999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.70299999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.52400000000000002}]},{"osd":1,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.55900000000000005,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.67100000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.54600000000000004,"15min":0.55300000000000005},"min":{"1min":0.34799999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.66200000000000003,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.32800000000000001}]},{"osd":2,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55000000000000004,"15min":0.55800000000000005},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":0.65800000000000003,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.55600000000000005,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66400000000000003,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Mar 21 12:32:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.54800000000000004,"15min":0.55900000000000005},"min":{"1min":0.39800000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.63200000000000001,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.56699999999999995,"15min":0.58099999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64700000000000002,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.51300000000000001}]},{"osd":5,"last update":"Thu Mar 21 12:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.29699999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.30399999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.34999999999999998}]},{"osd":6,"last update":"Thu Mar 21 12:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.498},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.371}]},{"osd":7,"last update":"Thu Mar 21 12:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47799999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.56399999999999995,"15min":0.57399999999999995},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68400000000000005,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.374},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.55400000000000005,"15min":0.56000000000000005},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66200000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.47299999999999998}]},{"osd":1,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.61499999999999999,"15min":0.57499999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":8.6940000000000008,"5min":8.6940000000000008,"15min":8.6940000000000008},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.36799999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.627,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.61199999999999999,"15min":0.56799999999999995},"min":{"1min":0.39600000000000002,"5min":0.379,"15min":0.379},"max":{"1min":8.6699999999999999,"5min":8.6699999999999999,"15min":8.6699999999999999},"last":1.7250000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70299999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.434}]},{"osd":3,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58399999999999996,"15min":0.58999999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.69499999999999995,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61399999999999999,"15min":0.623},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.71199999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.60499999999999998}]},{"osd":4,"last update":"Thu Mar 21 12:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.40999999999999998},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.622,"5min":0.75,"15min":0.75},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.45000000000000001,"15min":0.443},"min":{"1min":0.30099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42099999999999999}]},{"osd":6,"last update":"Thu Mar 21 12:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55300000000000005}]},{"osd":7,"last update":"Thu Mar 21 12:33:06 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.44700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T12:33:36.473 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T12:33:36.473 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T12:33:36.474 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-21T12:33:36.474 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-21T12:33:36.690 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T12:33:37.028 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:33:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:36] "GET /metrics HTTP/1.1" 200 191499 "" "Prometheus/2.33.4" 2024-03-21T12:33:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:37 smithi172 ceph-mon[24813]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:33:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:37 smithi172 ceph-mon[24813]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:37 smithi172 ceph-mon[29218]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:33:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:37 smithi172 ceph-mon[29218]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:37.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:37 smithi177 ceph-mon[25767]: from='client.24493 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T12:33:37.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:37 smithi177 ceph-mon[25767]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:38.429 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:33:38.429 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T12:33:39.169 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-21T12:33:39.169 INFO:tasks.cephadm:Setup complete, yielding 2024-03-21T12:33:39.169 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T12:33:39.179 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi172.front.sepia.ceph.com 2024-03-21T12:33:39.179 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-21T12:33:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:39 smithi172 ceph-mon[24813]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3128709756' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T12:33:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:39 smithi172 ceph-mon[29218]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3128709756' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T12:33:39.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:39 smithi177 ceph-mon[25767]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:39.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3128709756' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T12:33:41.130 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:33:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:33:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:41 smithi172 ceph-mon[29218]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:41 smithi172 ceph-mon[24813]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:41.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:41 smithi177 ceph-mon[25767]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:42.187 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:33:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42", 2024-03-21T12:33:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "r", 2024-03-21T12:33:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "current_period": "a55361d4-ad49-4a9a-a487-1c52960cfd28", 2024-03-21T12:33:42.187 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 1 2024-03-21T12:33:42.187 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:33:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:42 smithi177 ceph-mon[25767]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T12:33:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:42 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1110799743' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T12:33:42.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:42 smithi172 ceph-mon[24813]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T12:33:42.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:42 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1110799743' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T12:33:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:42 smithi172 ceph-mon[29218]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T12:33:42.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:42 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1110799743' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T12:33:42.833 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-21T12:33:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:43 smithi177 ceph-mon[25767]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:43 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1110799743' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T12:33:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:43 smithi177 ceph-mon[25767]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T12:33:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:43 smithi172 ceph-mon[24813]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:43 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1110799743' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T12:33:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:43 smithi172 ceph-mon[24813]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T12:33:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:43 smithi172 ceph-mon[29218]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:43 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1110799743' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T12:33:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:43 smithi172 ceph-mon[29218]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T12:33:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:44 smithi172 ceph-mon[24813]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T12:33:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:44 smithi172 ceph-mon[29218]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T12:33:44.518 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:44 smithi177 ceph-mon[25767]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "8280be28-afd6-48ad-9165-a6c73609f744", 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default", 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "default", 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": "true", 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [], 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "", 2024-03-21T12:33:44.640 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [], 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [], 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "", 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42", 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:44.641 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:33:44.858 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:44 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:33:45.160 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:44 smithi177 podman[41868]: 2024-03-21 12:33:44.812865016 +0000 UTC m=+0.018166683 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T12:33:45.161 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:44 smithi177 podman[41868]: 2024-03-21 12:33:44.919429643 +0000 UTC m=+0.124731372 container create 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, io.buildah.version=1.24.2, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, io.openshift.tags=base rhel8, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, version=8.5, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-21T12:33:45.276 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:45 smithi172 ceph-mon[24813]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:45.276 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:45 smithi172 ceph-mon[29218]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:45.303 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-21T12:33:45.412 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-mon[25767]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:33:45.412 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 podman[41868]: 2024-03-21 12:33:45.181287651 +0000 UTC m=+0.386589328 container init 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, com.redhat.component=ubi8-container, name=ubi8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, description=Ceph Grafana Container, vcs-type=git, io.openshift.expose-services=, release=236.1648460182) 2024-03-21T12:33:45.413 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 podman[41868]: 2024-03-21 12:33:45.184018684 +0000 UTC m=+0.389320352 container start 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, io.openshift.expose-services=, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, version=8.5, name=ubi8, vcs-type=git, architecture=x86_64) 2024-03-21T12:33:45.413 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-21T12:33:45.413 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-21T12:33:45.413 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-21T12:33:45.413 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-21T12:33:45.413 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-21T12:33:45.414 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="App mode production" logger=settings 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-21T12:33:45.415 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-21T12:33:45.416 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-21T12:33:45.417 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 bash[41868]: 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-21T12:33:45.418 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-21T12:33:45.419 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-21T12:33:45.419 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-21T12:33:45.419 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-21T12:33:45.419 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-21T12:33:45.419 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-21T12:33:45.419 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-21T12:33:45.420 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-21T12:33:45.421 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-21T12:33:45.422 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-21T12:33:45.423 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-21T12:33:45.424 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-21T12:33:45.425 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-21T12:33:45.426 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-21T12:33:45.427 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-21T12:33:45.427 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-21T12:33:45.665 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-21T12:33:45.665 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-21T12:33:45.665 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-21T12:33:45.665 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-21T12:33:45.666 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-21T12:33:45.667 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-21T12:33:45.667 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-21T12:33:45.667 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-21T12:33:45.667 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-21T12:33:45.667 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-21T12:33:45.668 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-21T12:33:45.669 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-21T12:33:45.670 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-21T12:33:45.671 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-21T12:33:45.672 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-21T12:33:45.673 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-21T12:33:45.674 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-21T12:33:45.675 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-21T12:33:45.676 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-21T12:33:45.677 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-21T12:33:45.678 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-21T12:33:45.679 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-21T12:33:45.680 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-21T12:33:45.681 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-21T12:33:46.038 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-21T12:33:46.039 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-21T12:33:46.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-21T12:33:46.041 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-21T12:33:46.042 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=510.897067ms 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T12:33:46.043 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:33:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:33:45+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[24813]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:46.516 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:33:46.517 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:33:46.517 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:33:46.517 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-mon[29218]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-21T12:33:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:46 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:46 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:33:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:46 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:33:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:46 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:33:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:46 smithi177 ceph-mon[25767]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-21T12:33:46.836 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:33:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:46] "GET /metrics HTTP/1.1" 200 191502 "" "Prometheus/2.33.4" 2024-03-21T12:33:47.018 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:33:47.018 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "84112654-d4ae-438b-8e17-1d9c8ca8b932", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "z", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "domain_root": "z.rgw.meta:root", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "control_pool": "z.rgw.control", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "log_pool": "z.rgw.log", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-21T12:33:47.019 INFO:teuthology.orchestra.run.smithi172.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "otp_pool": "z.rgw.otp", 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "system_key": { 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "access_key": "", 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "secret_key": "" 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_pools": [ 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "default-placement", 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "val": { 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": { 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD": { 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:47.020 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: "index_type": 0 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42", 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-21T12:33:47.021 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:33:47.604 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-21T12:33:49.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:49 smithi172 ceph-mon[24813]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-21T12:33:49.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:49 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:49.208 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:49 smithi172 ceph-mon[29218]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-21T12:33:49.208 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:49 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:49.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:49 smithi177 ceph-mon[25767]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-21T12:33:49.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:49 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:50 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:50 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:50.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:50 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:51.162 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:33:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[24813]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[24813]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1402969751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[29218]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[29218]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T12:33:51.463 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1402969751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:33:51.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:51 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:33:51.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:51 smithi177 ceph-mon[25767]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-21T12:33:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:51 smithi177 ceph-mon[25767]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T12:33:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:51 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1402969751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:33:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:51 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-21T12:33:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-21T12:33:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[24813]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T12:33:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-21T12:33:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[29218]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T12:33:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:52 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:52 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-21T12:33:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:52 smithi177 ceph-mon[25767]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T12:33:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:52 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:52 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:53.178 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: Reconfiguring daemon alertmanager.a on smithi172 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1402969751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-21T12:33:53.179 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[24813]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T12:33:53.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T12:33:53.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: Reconfiguring daemon alertmanager.a on smithi172 2024-03-21T12:33:53.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-21T12:33:53.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T12:33:53.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1402969751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:33:53.606 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:33:53.607 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-21T12:33:53.607 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:53 smithi172 ceph-mon[29218]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T12:33:53.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: Reconfiguring daemon alertmanager.a on smithi172 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: pgmap v52: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1402969751' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-21T12:33:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:53 smithi177 ceph-mon[25767]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T12:33:54.848 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:54 smithi172 systemd[1]: Stopping Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:33:54.848 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:54 smithi172 bash[49135]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T12:33:55.132 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[46806]: level=info ts=2024-03-21T12:33:54.848Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:33:55.132 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:54 smithi172 podman[49148]: 2024-03-21 12:33:54.869071672 +0000 UTC m=+0.118602525 container died 5bee4969042066f3ac966b7d5268bdc972d3bda9f6dffdb5067d3853ab8c8eda (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:33:55.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 podman[49148]: 2024-03-21 12:33:55.231659266 +0000 UTC m=+0.481190125 container cleanup 5bee4969042066f3ac966b7d5268bdc972d3bda9f6dffdb5067d3853ab8c8eda (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:33:55.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 bash[49148]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T12:33:55.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 bash[49177]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T12:33:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[29218]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:33:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[29218]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[24813]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[24813]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:33:55.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:55 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:33:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:55 smithi177 ceph-mon[25767]: pgmap v55: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:33:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:55 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:55 smithi177 ceph-mon[25767]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T12:33:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:55 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:33:55.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:55 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T12:33:55.825 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 podman[49162]: 2024-03-21 12:33:55.533040534 +0000 UTC m=+0.664104020 container remove 5bee4969042066f3ac966b7d5268bdc972d3bda9f6dffdb5067d3853ab8c8eda (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:33:55.825 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 podman[49162]: 2024-03-21 12:33:55.608611587 +0000 UTC m=+0.739675084 volume remove 13eb3fac86193f2b428a4b5a7a5c215cc8a899ec3b2f1f021611019fae79fc32 2024-03-21T12:33:55.825 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T12:33:55.825 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 systemd[1]: Stopped Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:33:55.825 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 systemd[1]: Starting Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:33:56.089 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 podman[49244]: 2024-03-21 12:33:55.825906809 +0000 UTC m=+0.105348220 volume create 6a3fed3d06f604239c247e2d57a82ed17b99463450ef20510200aebabd2861d9 2024-03-21T12:33:56.089 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 podman[49244]: 2024-03-21 12:33:55.738602538 +0000 UTC m=+0.018043962 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-21T12:33:56.089 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:55 smithi172 podman[49244]: 2024-03-21 12:33:55.926302696 +0000 UTC m=+0.205744108 container create c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:33:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:56 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:33:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:56 smithi177 ceph-mon[25767]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T12:33:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:56 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:33:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:56 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:33:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:33:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[24813]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T12:33:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:33:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:33:56.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-21T12:33:56.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[29218]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T12:33:56.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:33:56.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 podman[49244]: 2024-03-21 12:33:56.19636565 +0000 UTC m=+0.475807071 container init c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 podman[49244]: 2024-03-21 12:33:56.199179417 +0000 UTC m=+0.478620834 container start c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.209Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.209Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.211Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.172 port=9094 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.212Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T12:33:56.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.236Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T12:33:56.648 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.237Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T12:33:56.648 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.239Z caller=main.go:518 msg=Listening address=:9093 2024-03-21T12:33:56.648 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:56.239Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-21T12:33:56.648 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 bash[49244]: c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a 2024-03-21T12:33:56.648 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:56 smithi172 systemd[1]: Started Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:33:57.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:33:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:33:56] "GET /metrics HTTP/1.1" 200 195361 "" "Prometheus/2.33.4" 2024-03-21T12:33:57.234 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:33:57.234 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "8998f00c-e9d0-4723-a327-c957f9a692b7", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "epoch": 1, 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "predecessor_uuid": "a55361d4-ad49-4a9a-a487-1c52960cfd28", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_status": [], 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "period_map": { 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "8998f00c-e9d0-4723-a327-c957f9a692b7", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "zonegroups": [ 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "8280be28-afd6-48ad-9165-a6c73609f744", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "api_name": "default", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "is_master": "true", 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [], 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames": [], 2024-03-21T12:33:57.235 INFO:teuthology.orchestra.run.smithi172.stdout: "hostnames_s3website": [], 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "84112654-d4ae-438b-8e17-1d9c8ca8b932", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "zones": [ 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "id": "84112654-d4ae-438b-8e17-1d9c8ca8b932", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "z", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "endpoints": [], 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "log_meta": "false", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "log_data": "false", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_index_max_shards": 11, 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "read_only": "false", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "tier_type": "", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from_all": "true", 2024-03-21T12:33:57.236 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_from": [], 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "redirect_zone": "" 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "placement_targets": [ 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "name": "default-placement", 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "tags": [], 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "storage_classes": [ 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "STANDARD" 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "default_placement": "default-placement", 2024-03-21T12:33:57.237 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42", 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "sync_policy": { 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "groups": [] 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "short_zone_ids": [ 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: { 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "key": "84112654-d4ae-438b-8e17-1d9c8ca8b932", 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "val": 779206747 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:57.238 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zonegroup": "8280be28-afd6-48ad-9165-a6c73609f744", 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "master_zone": "84112654-d4ae-438b-8e17-1d9c8ca8b932", 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "period_config": { 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_quota": { 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "user_quota": { 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "check_on_raw": false, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size": -1, 2024-03-21T12:33:57.239 INFO:teuthology.orchestra.run.smithi172.stdout: "max_size_kb": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_objects": -1 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "user_ratelimit": { 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "bucket_ratelimit": { 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2024-03-21T12:33:57.240 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "anonymous_ratelimit": { 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_ops": 0, 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_ops": 0, 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "max_read_bytes": 0, 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "max_write_bytes": 0, 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "enabled": false 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_id": "fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42", 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_name": "r", 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout: "realm_epoch": 2 2024-03-21T12:33:57.241 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:33:57.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:57 smithi177 ceph-mon[25767]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T12:33:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:57 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:33:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:57 smithi177 ceph-mon[25767]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T12:33:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:57 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T12:33:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:57 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T12:33:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:57 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:57.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[24813]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T12:33:57.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:33:57.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[24813]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T12:33:57.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T12:33:57.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T12:33:57.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:57.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[29218]: pgmap v58: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T12:33:57.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T12:33:57.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[29218]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T12:33:57.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/922776969' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T12:33:57.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-21T12:33:57.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:57 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:33:57.929 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-21T12:33:58.187 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:33:58.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:58 smithi177 ceph-mon[25767]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:33:58.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:58 smithi177 ceph-mon[25767]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:33:58.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:58 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-21T12:33:58.279 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:58 smithi177 ceph-mon[25767]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T12:33:58.280 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:33:58 smithi177 ceph-mon[25767]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T12:33:58.463 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:33:58.213Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000463379s 2024-03-21T12:33:58.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:33:58.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-21T12:33:58.463 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[24813]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T12:33:58.464 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[24813]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T12:33:58.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[29218]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:33:58.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[29218]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:33:58.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-21T12:33:58.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[29218]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T12:33:58.464 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:33:58 smithi172 ceph-mon[29218]: pgmap v61: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-21T12:33:59.459 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:33:59.459 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 bash[42568]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:33:59.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.458Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.459Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[40907]: ts=2024-03-21T12:33:59.460Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T12:33:59.751 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 podman[42579]: 2024-03-21 12:33:59.469030961 +0000 UTC m=+0.130427095 container died ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:34:00.006 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled rgw.foo update... 2024-03-21T12:34:00.107 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 podman[42579]: 2024-03-21 12:33:59.841510027 +0000 UTC m=+0.502906198 container cleanup ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:34:00.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 bash[42579]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T12:34:00.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:33:59 smithi177 bash[42607]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:34:00.446 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 podman[42593]: 2024-03-21 12:34:00.108992931 +0000 UTC m=+0.640954147 container remove ccac4109bec5439bd882f1d5031c69eb72c7776bcf25fb87015fefdb4f97336a (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:34:00.446 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T12:34:00.446 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 systemd[1]: Stopped Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:34:00.446 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:34:00.446 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 podman[42677]: 2024-03-21 12:34:00.249687935 +0000 UTC m=+0.020025056 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T12:34:00.447 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 podman[42677]: 2024-03-21 12:34:00.351567747 +0000 UTC m=+0.121904859 container create 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:34:00.717 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph osd pool create foo' 2024-03-21T12:34:00.798 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 podman[42677]: 2024-03-21 12:34:00.545425619 +0000 UTC m=+0.315762730 container init 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:34:00.798 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 podman[42677]: 2024-03-21 12:34:00.547961893 +0000 UTC m=+0.318299016 container start 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:34:00.798 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.570Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T12:34:00.798 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.570Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T12:34:00.798 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.570Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T12:34:00.798 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.570Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.570Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.570Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.571Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.572Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.573Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.574Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.574Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.451µs 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.574Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 bash[42677]: 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.781Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.781Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.782Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.954µs wal_replay_duration=207.242983ms total_replay_duration=207.27732ms 2024-03-21T12:34:00.799 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.783Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:34:00.800 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.783Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T12:34:00.800 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.783Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:34:00.800 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.798Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.931329ms db_storage=678ns remote_storage=1.199µs web_handler=381ns query_engine=694ns scrape=345.203µs scrape_sd=22.652µs notify=19.78µs notify_sd=7.494µs rules=14.252867ms 2024-03-21T12:34:00.800 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:34:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:34:00.798Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: Saving service rgw.foo spec with placement count:2 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: pgmap v62: 129 pgs: 15 creating+peering, 114 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.172:9093"}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.177:9095"}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.177:3000"}]: dispatch 2024-03-21T12:34:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:01 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: Saving service rgw.foo spec with placement count:2 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: pgmap v62: 129 pgs: 15 creating+peering, 114 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.172:9093"}]: dispatch 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:34:01.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.177:9095"}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.177:3000"}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: Saving service rgw.foo spec with placement count:2 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.285 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: pgmap v62: 129 pgs: 15 creating+peering, 114 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.172:9093"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.177:9095"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.177:3000"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:01.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:01 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:02.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:02 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:34:02.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:02 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.172:9093"}]: dispatch 2024-03-21T12:34:02.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:02 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:34:02.267 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:02 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.177:9095"}]: dispatch 2024-03-21T12:34:02.268 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:02 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:34:02.268 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:02 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.177:3000"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.172:9093"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.177:9095"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.177:3000"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.172:9093"}]: dispatch 2024-03-21T12:34:02.272 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:34:02.273 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.177:9095"}]: dispatch 2024-03-21T12:34:02.273 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:34:02.273 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:02 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.177:3000"}]: dispatch 2024-03-21T12:34:03.028 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'foo' created 2024-03-21T12:34:03.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:03 smithi177 ceph-mon[25767]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-21T12:34:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:03 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/859399076' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T12:34:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:03 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T12:34:03.217 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:03 smithi172 ceph-mon[24813]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-21T12:34:03.217 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:03 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/859399076' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T12:34:03.217 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:03 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T12:34:03.217 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:03 smithi172 ceph-mon[29218]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-21T12:34:03.217 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:03 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/859399076' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T12:34:03.217 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:03 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-21T12:34:03.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'rbd pool init foo' 2024-03-21T12:34:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:04 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-21T12:34:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:04 smithi172 ceph-mon[24813]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T12:34:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:04 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-21T12:34:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:04 smithi172 ceph-mon[29218]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T12:34:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:04 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-21T12:34:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:04 smithi177 ceph-mon[25767]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T12:34:05.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[24813]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T12:34:05.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:05.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[24813]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-21T12:34:05.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[24813]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T12:34:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[29218]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T12:34:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[29218]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-21T12:34:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:05 smithi172 ceph-mon[29218]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T12:34:05.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:05 smithi177 ceph-mon[25767]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T12:34:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:05 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:05 smithi177 ceph-mon[25767]: pgmap v66: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-21T12:34:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:05 smithi177 ceph-mon[25767]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T12:34:06.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:06 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/67274158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T12:34:06.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:06 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T12:34:06.302 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:06 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/67274158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T12:34:06.302 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:06 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T12:34:06.302 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:34:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:34:06.215Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003110183s 2024-03-21T12:34:06.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:06 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/67274158' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T12:34:06.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:06 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-21T12:34:06.935 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:34:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:06] "GET /metrics HTTP/1.1" 200 195361 "" "Prometheus/2.33.4" 2024-03-21T12:34:07.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:07 smithi172 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-21T12:34:07.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:07 smithi172 ceph-mon[24813]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T12:34:07.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:07 smithi172 ceph-mon[24813]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:34:07.205 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:07 smithi172 ceph-mon[29218]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-21T12:34:07.205 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:07 smithi172 ceph-mon[29218]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T12:34:07.205 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:07 smithi172 ceph-mon[29218]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:34:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:07 smithi177 ceph-mon[25767]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-21T12:34:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:07 smithi177 ceph-mon[25767]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T12:34:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:07 smithi177 ceph-mon[25767]: pgmap v69: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:08.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:08.254 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:34:08.254 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.254 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:08 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:34:08.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:08.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:08 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:08.914 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[24813]: Saving service rgw.foo spec with placement count:2 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[24813]: Deploying daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[24813]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[24813]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[29218]: Saving service rgw.foo spec with placement count:2 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[29218]: Deploying daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T12:34:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[29218]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T12:34:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:09 smithi172 ceph-mon[29218]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:34:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:09 smithi177 ceph-mon[25767]: Saving service rgw.foo spec with placement count:2 2024-03-21T12:34:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:09 smithi177 ceph-mon[25767]: Deploying daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T12:34:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:09 smithi177 ceph-mon[25767]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T12:34:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:09 smithi177 ceph-mon[25767]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:34:11.132 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:34:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:34:11.264 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled iscsi.foo update... 2024-03-21T12:34:11.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:11 smithi172 ceph-mon[24813]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 405 B/s wr, 0 op/s 2024-03-21T12:34:11.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:11 smithi172 ceph-mon[29218]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 405 B/s wr, 0 op/s 2024-03-21T12:34:11.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:11 smithi177 ceph-mon[25767]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 405 B/s wr, 0 op/s 2024-03-21T12:34:11.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 180' 2024-03-21T12:34:12.457 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:34:12.457 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: Saving service iscsi.foo spec with placement count:1 2024-03-21T12:34:12.457 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 339 B/s wr, 0 op/s 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: Saving service iscsi.foo spec with placement count:1 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.458 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 339 B/s wr, 0 op/s 2024-03-21T12:34:12.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:12.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:12.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:34:12.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.459 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:12 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:12.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: Saving service iscsi.foo spec with placement count:1 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 339 B/s wr, 0 op/s 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:12 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:13.624 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:13 smithi172 ceph-mon[24813]: Deploying daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T12:34:13.624 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:13 smithi172 ceph-mon[29218]: Deploying daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T12:34:13.840 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:13 smithi177 ceph-mon[25767]: Deploying daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T12:34:14.837 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:14 smithi177 ceph-mon[25767]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T12:34:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:14 smithi172 ceph-mon[24813]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T12:34:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:14 smithi172 ceph-mon[29218]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-21T12:34:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:16 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:16 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:16 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:16 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:16 smithi177 ceph-mon[25767]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 124 op/s 2024-03-21T12:34:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:16 smithi177 ceph-mon[25767]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:16] "GET /metrics HTTP/1.1" 200 197930 "" "Prometheus/2.33.4" 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[24813]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 124 op/s 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[24813]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T12:34:17.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[29218]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.9 KiB/s wr, 124 op/s 2024-03-21T12:34:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:16 smithi172 ceph-mon[29218]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T12:34:19.374 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:19 smithi177 ceph-mon[25767]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-21T12:34:19.374 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:19 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:19 smithi172 ceph-mon[24813]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-21T12:34:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:19 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:19 smithi172 ceph-mon[29218]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-21T12:34:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:19 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:34:20.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:34:20.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T12:34:20.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:20 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T12:34:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:21.164 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:34:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[24813]: Checking dashboard <-> RGW credentials 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[24813]: Checking pool "foo" exists for service iscsi.foo 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[24813]: Deploying daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[24813]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 3.5 KiB/s wr, 191 op/s 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[29218]: Checking dashboard <-> RGW credentials 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[29218]: Checking pool "foo" exists for service iscsi.foo 2024-03-21T12:34:21.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[29218]: Deploying daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:34:21.385 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:21 smithi172 ceph-mon[29218]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 3.5 KiB/s wr, 191 op/s 2024-03-21T12:34:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:21 smithi177 ceph-mon[25767]: Checking dashboard <-> RGW credentials 2024-03-21T12:34:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:21 smithi177 ceph-mon[25767]: Checking pool "foo" exists for service iscsi.foo 2024-03-21T12:34:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:21 smithi177 ceph-mon[25767]: Deploying daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:34:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:21 smithi177 ceph-mon[25767]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 56 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 3.5 KiB/s wr, 191 op/s 2024-03-21T12:34:22.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:22 smithi172 ceph-mon[24813]: mgrmap e20: y(active, since 117s), standbys: x 2024-03-21T12:34:22.337 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:22 smithi172 ceph-mon[29218]: mgrmap e20: y(active, since 117s), standbys: x 2024-03-21T12:34:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:22 smithi177 ceph-mon[25767]: mgrmap e20: y(active, since 117s), standbys: x 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[24813]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.6 KiB/s wr, 216 op/s 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/998865484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[29218]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.6 KiB/s wr, 216 op/s 2024-03-21T12:34:23.410 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:23 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/998865484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:34:23.478 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:23 smithi177 ceph-mon[25767]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.6 KiB/s wr, 216 op/s 2024-03-21T12:34:23.478 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:23 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:23.478 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:23 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:23.478 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:23 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:23.478 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:23 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:23.478 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:23 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/998865484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2448771936' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4048413631"}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:34:24.512 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2448771936' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4048413631"}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:34:24.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:34:24.514 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:34:24.514 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:34:24.514 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:24.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2448771936' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4048413631"}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:34:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.6 KiB/s wr, 216 op/s 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2448771936' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4048413631"}]': finished 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T12:34:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/459595801' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/238925532"}]: dispatch 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.6 KiB/s wr, 216 op/s 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2448771936' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4048413631"}]': finished 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T12:34:25.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:25 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/459595801' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/238925532"}]: dispatch 2024-03-21T12:34:25.453 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 57 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 3.6 KiB/s wr, 216 op/s 2024-03-21T12:34:25.453 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2448771936' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4048413631"}]': finished 2024-03-21T12:34:25.453 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-03-21T12:34:25.453 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-03-21T12:34:25.453 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-03-21T12:34:25.454 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-21T12:34:25.454 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T12:34:25.454 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:25 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/459595801' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/238925532"}]: dispatch 2024-03-21T12:34:26.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:26 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/459595801' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/238925532"}]': finished 2024-03-21T12:34:26.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:26 smithi177 ceph-mon[25767]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T12:34:26.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:26 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/425536667' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3682709254"}]: dispatch 2024-03-21T12:34:26.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:26 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/459595801' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/238925532"}]': finished 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[24813]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/425536667' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3682709254"}]: dispatch 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/459595801' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/238925532"}]': finished 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[29218]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/425536667' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3682709254"}]: dispatch 2024-03-21T12:34:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:34:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:26] "GET /metrics HTTP/1.1" 200 208095 "" "Prometheus/2.33.4" 2024-03-21T12:34:27.473 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 511 B/s wr, 132 op/s 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/425536667' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3682709254"}]': finished 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: Adding iSCSI gateway http://:@172.21.15.172:5000 to Dashboard 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi172"}]: dispatch 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi172"}]: dispatch 2024-03-21T12:34:27.474 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: Checking dashboard <-> RGW credentials 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1313937478' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/1178568884"}]: dispatch 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 511 B/s wr, 132 op/s 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/425536667' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3682709254"}]': finished 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.475 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: Adding iSCSI gateway http://:@172.21.15.172:5000 to Dashboard 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi172"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi172"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: Checking dashboard <-> RGW credentials 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.476 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:27 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1313937478' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/1178568884"}]: dispatch 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 511 B/s wr, 132 op/s 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/425536667' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3682709254"}]': finished 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: Adding iSCSI gateway http://:@172.21.15.172:5000 to Dashboard 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi172"}]: dispatch 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi172"}]: dispatch 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: Checking dashboard <-> RGW credentials 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:34:27.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:27 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1313937478' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/1178568884"}]: dispatch 2024-03-21T12:34:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:28 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1313937478' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/1178568884"}]': finished 2024-03-21T12:34:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:28 smithi177 ceph-mon[25767]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T12:34:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:28 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3292642390' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]: dispatch 2024-03-21T12:34:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:28 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]: dispatch 2024-03-21T12:34:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:28 smithi177 ceph-mon[25767]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-21T12:34:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1313937478' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/1178568884"}]': finished 2024-03-21T12:34:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[24813]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T12:34:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3292642390' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]: dispatch 2024-03-21T12:34:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]: dispatch 2024-03-21T12:34:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[24813]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-21T12:34:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1313937478' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/1178568884"}]': finished 2024-03-21T12:34:28.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[29218]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T12:34:28.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3292642390' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]: dispatch 2024-03-21T12:34:28.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]: dispatch 2024-03-21T12:34:28.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:28 smithi172 ceph-mon[29218]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 2 op/s 2024-03-21T12:34:29.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:29 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]': finished 2024-03-21T12:34:29.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:29 smithi177 ceph-mon[25767]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T12:34:29.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:29 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3176878255' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]: dispatch 2024-03-21T12:34:29.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:29 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]: dispatch 2024-03-21T12:34:29.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]': finished 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[24813]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3176878255' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]: dispatch 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]: dispatch 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/1178568884"}]': finished 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[29218]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3176878255' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]: dispatch 2024-03-21T12:34:29.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:29 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]: dispatch 2024-03-21T12:34:30.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:30 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]': finished 2024-03-21T12:34:30.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:30 smithi177 ceph-mon[25767]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T12:34:30.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:30 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3841248277' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1643456920"}]: dispatch 2024-03-21T12:34:30.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:30 smithi177 ceph-mon[25767]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 11 B/s, 0 objects/s recovering 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]': finished 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[24813]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3841248277' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1643456920"}]: dispatch 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[24813]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 11 B/s, 0 objects/s recovering 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3682709254"}]': finished 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[29218]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T12:34:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3841248277' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1643456920"}]: dispatch 2024-03-21T12:34:30.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:30 smithi172 ceph-mon[29218]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 8.7 KiB/s rd, 8 op/s; 11 B/s, 0 objects/s recovering 2024-03-21T12:34:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:34:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:34:31.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:31 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3841248277' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1643456920"}]': finished 2024-03-21T12:34:31.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:31 smithi177 ceph-mon[25767]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T12:34:31.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:31 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3504024140' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3553472034"}]: dispatch 2024-03-21T12:34:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:31 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3841248277' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1643456920"}]': finished 2024-03-21T12:34:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:31 smithi172 ceph-mon[24813]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T12:34:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:31 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3504024140' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3553472034"}]: dispatch 2024-03-21T12:34:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:31 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3841248277' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1643456920"}]': finished 2024-03-21T12:34:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:31 smithi172 ceph-mon[29218]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T12:34:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:31 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3504024140' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3553472034"}]: dispatch 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3504024140' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3553472034"}]': finished 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[24813]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3829214156' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4216383095"}]: dispatch 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[24813]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3504024140' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3553472034"}]': finished 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[29218]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T12:34:32.844 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3829214156' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4216383095"}]: dispatch 2024-03-21T12:34:32.845 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:32 smithi172 ceph-mon[29218]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-21T12:34:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:32 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3504024140' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3553472034"}]': finished 2024-03-21T12:34:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:32 smithi177 ceph-mon[25767]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T12:34:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:32 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3829214156' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4216383095"}]: dispatch 2024-03-21T12:34:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:32 smithi177 ceph-mon[25767]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-21T12:34:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:33 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3829214156' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4216383095"}]': finished 2024-03-21T12:34:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:33 smithi177 ceph-mon[25767]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T12:34:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:33 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1596432002' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]: dispatch 2024-03-21T12:34:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:33 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]: dispatch 2024-03-21T12:34:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3829214156' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4216383095"}]': finished 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[24813]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1596432002' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]: dispatch 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]: dispatch 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3829214156' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4216383095"}]': finished 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[29218]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1596432002' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]: dispatch 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]: dispatch 2024-03-21T12:34:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:34 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]': finished 2024-03-21T12:34:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:34 smithi177 ceph-mon[25767]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T12:34:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:34 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3904609037' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]: dispatch 2024-03-21T12:34:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:34 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]: dispatch 2024-03-21T12:34:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:34 smithi177 ceph-mon[25767]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]': finished 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[24813]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3904609037' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]: dispatch 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]: dispatch 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[24813]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3623707692"}]': finished 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[29218]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3904609037' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]: dispatch 2024-03-21T12:34:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]: dispatch 2024-03-21T12:34:34.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:34 smithi172 ceph-mon[29218]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T12:34:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:35 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]': finished 2024-03-21T12:34:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:35 smithi177 ceph-mon[25767]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T12:34:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:35 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1491603471' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4198175409"}]: dispatch 2024-03-21T12:34:35.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:35 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]': finished 2024-03-21T12:34:35.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:35 smithi172 ceph-mon[24813]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T12:34:35.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:35 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1491603471' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4198175409"}]: dispatch 2024-03-21T12:34:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:35 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/422523718"}]': finished 2024-03-21T12:34:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:35 smithi172 ceph-mon[29218]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T12:34:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:35 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1491603471' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4198175409"}]: dispatch 2024-03-21T12:34:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:36 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1491603471' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4198175409"}]': finished 2024-03-21T12:34:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:36 smithi177 ceph-mon[25767]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T12:34:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:36 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/803275381' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3189318036"}]: dispatch 2024-03-21T12:34:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:36 smithi177 ceph-mon[25767]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:36.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:36] "GET /metrics HTTP/1.1" 200 208095 "" "Prometheus/2.33.4" 2024-03-21T12:34:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1491603471' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4198175409"}]': finished 2024-03-21T12:34:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[24813]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T12:34:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/803275381' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3189318036"}]: dispatch 2024-03-21T12:34:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[24813]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1491603471' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4198175409"}]': finished 2024-03-21T12:34:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[29218]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T12:34:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/803275381' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3189318036"}]: dispatch 2024-03-21T12:34:36.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:36 smithi172 ceph-mon[29218]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:37 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/803275381' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3189318036"}]': finished 2024-03-21T12:34:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:37 smithi177 ceph-mon[25767]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T12:34:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:37 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1020268134' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1465987532"}]: dispatch 2024-03-21T12:34:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:37 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/803275381' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3189318036"}]': finished 2024-03-21T12:34:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:37 smithi172 ceph-mon[29218]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T12:34:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:37 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1020268134' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1465987532"}]: dispatch 2024-03-21T12:34:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:37 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/803275381' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3189318036"}]': finished 2024-03-21T12:34:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:37 smithi172 ceph-mon[24813]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T12:34:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:37 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1020268134' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1465987532"}]: dispatch 2024-03-21T12:34:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:38 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1020268134' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1465987532"}]': finished 2024-03-21T12:34:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:38 smithi177 ceph-mon[25767]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T12:34:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:38 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/752878943' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]: dispatch 2024-03-21T12:34:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:38 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]: dispatch 2024-03-21T12:34:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:38 smithi177 ceph-mon[25767]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1020268134' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1465987532"}]': finished 2024-03-21T12:34:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[24813]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T12:34:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/752878943' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]: dispatch 2024-03-21T12:34:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]: dispatch 2024-03-21T12:34:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[24813]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1020268134' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1465987532"}]': finished 2024-03-21T12:34:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[29218]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T12:34:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/752878943' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]: dispatch 2024-03-21T12:34:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]: dispatch 2024-03-21T12:34:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:38 smithi172 ceph-mon[29218]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 62 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]': finished 2024-03-21T12:34:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:39 smithi177 ceph-mon[25767]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T12:34:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/990715336' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]: dispatch 2024-03-21T12:34:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]: dispatch 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]': finished 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[24813]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/990715336' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]: dispatch 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]: dispatch 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3168760134"}]': finished 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[29218]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/990715336' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]: dispatch 2024-03-21T12:34:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]: dispatch 2024-03-21T12:34:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:40 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]': finished 2024-03-21T12:34:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:40 smithi177 ceph-mon[25767]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T12:34:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:40 smithi177 ceph-mon[25767]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:40 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]': finished 2024-03-21T12:34:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:40 smithi172 ceph-mon[24813]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T12:34:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:40 smithi172 ceph-mon[24813]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:40 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3189318036"}]': finished 2024-03-21T12:34:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:40 smithi172 ceph-mon[29218]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T12:34:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:40 smithi172 ceph-mon[29218]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:41.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:34:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:34:42.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:42 smithi172 ceph-mon[24813]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T12:34:42.854 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:42 smithi172 ceph-mon[29218]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T12:34:42.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:42 smithi177 ceph-mon[25767]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T12:34:43.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:44.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:44 smithi177 ceph-mon[25767]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:34:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:44 smithi172 ceph-mon[24813]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:34:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:44 smithi172 ceph-mon[29218]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:34:47.142 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:34:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:46] "GET /metrics HTTP/1.1" 200 208102 "" "Prometheus/2.33.4" 2024-03-21T12:34:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:47 smithi172 ceph-mon[24813]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:47 smithi172 ceph-mon[29218]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:47.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:47 smithi177 ceph-mon[25767]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:49 smithi172 ceph-mon[24813]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:34:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:49 smithi172 ceph-mon[29218]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:34:49.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:49 smithi177 ceph-mon[25767]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:34:51.149 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:34:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:34:51.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:51 smithi177 ceph-mon[25767]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:34:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:51 smithi172 ceph-mon[24813]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:34:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:51 smithi172 ceph-mon[29218]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:34:52.614 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:52 smithi172 ceph-mon[24813]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:52.614 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:52 smithi172 ceph-mon[29218]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:52.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:52 smithi177 ceph-mon[25767]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:53.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:53.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:34:54.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:54 smithi177 ceph-mon[25767]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:54 smithi172 ceph-mon[24813]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:54 smithi172 ceph-mon[29218]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:57.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:34:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:34:56] "GET /metrics HTTP/1.1" 200 208064 "" "Prometheus/2.33.4" 2024-03-21T12:34:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:57 smithi172 ceph-mon[24813]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:57 smithi172 ceph-mon[29218]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:57.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:57 smithi177 ceph-mon[25767]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:34:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:34:59 smithi177 ceph-mon[25767]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:34:59 smithi172 ceph-mon[24813]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:34:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:34:59 smithi172 ceph-mon[29218]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:01.151 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:35:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:35:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:01 smithi177 ceph-mon[25767]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:01.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:01 smithi172 ceph-mon[24813]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:01 smithi172 ceph-mon[29218]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:02.852 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:02 smithi172 ceph-mon[24813]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:02.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:02 smithi172 ceph-mon[29218]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:02 smithi177 ceph-mon[25767]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:03 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:03 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:03 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:04.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:04 smithi177 ceph-mon[25767]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:04 smithi172 ceph-mon[24813]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:04 smithi172 ceph-mon[29218]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:07.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:35:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:06] "GET /metrics HTTP/1.1" 200 208064 "" "Prometheus/2.33.4" 2024-03-21T12:35:07.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:07 smithi177 ceph-mon[25767]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:07 smithi172 ceph-mon[24813]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:07 smithi172 ceph-mon[29218]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:09.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:09 smithi177 ceph-mon[25767]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:09.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:09 smithi172 ceph-mon[24813]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:09 smithi172 ceph-mon[29218]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:11.155 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:35:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:35:11.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:11 smithi177 ceph-mon[25767]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:11 smithi172 ceph-mon[24813]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:11 smithi172 ceph-mon[29218]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:12.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:12 smithi177 ceph-mon[25767]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:12.870 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:12 smithi172 ceph-mon[24813]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:12.870 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:12 smithi172 ceph-mon[29218]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:14.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:14 smithi177 ceph-mon[25767]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:14 smithi172 ceph-mon[24813]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:14 smithi172 ceph-mon[29218]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:17.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:35:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:16] "GET /metrics HTTP/1.1" 200 208029 "" "Prometheus/2.33.4" 2024-03-21T12:35:17.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:17 smithi177 ceph-mon[25767]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:17.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:17 smithi172 ceph-mon[24813]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:17 smithi172 ceph-mon[29218]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:19.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:19 smithi177 ceph-mon[25767]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:19.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:19 smithi172 ceph-mon[24813]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:19 smithi172 ceph-mon[29218]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:21.157 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:35:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:35:21.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:21 smithi172 ceph-mon[24813]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:21.449 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:21 smithi172 ceph-mon[29218]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:21 smithi177 ceph-mon[25767]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:22 smithi177 ceph-mon[25767]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:22.881 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:22 smithi172 ceph-mon[24813]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:22.881 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:22 smithi172 ceph-mon[29218]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:23.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:23 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:23 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:23 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:24 smithi177 ceph-mon[25767]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:35:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:35:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:35:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[24813]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[29218]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:35:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:35:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:35:26.775 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:35:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:26] "GET /metrics HTTP/1.1" 200 208016 "" "Prometheus/2.33.4" 2024-03-21T12:35:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:27 smithi177 ceph-mon[25767]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:35:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:35:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:35:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:35:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[29218]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:35:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:35:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:35:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:35:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[24813]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:35:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:35:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:35:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:35:29.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:29 smithi177 ceph-mon[25767]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:29.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:29 smithi172 ceph-mon[24813]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:29 smithi172 ceph-mon[29218]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:31.162 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:35:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:35:31.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:31 smithi177 ceph-mon[25767]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:31 smithi172 ceph-mon[24813]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:31 smithi172 ceph-mon[29218]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:32 smithi177 ceph-mon[25767]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:32.891 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:32 smithi172 ceph-mon[24813]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:32.891 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:32 smithi172 ceph-mon[29218]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:34.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:34 smithi177 ceph-mon[25767]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:34 smithi172 ceph-mon[24813]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:34 smithi172 ceph-mon[29218]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:37.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:35:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:36] "GET /metrics HTTP/1.1" 200 208016 "" "Prometheus/2.33.4" 2024-03-21T12:35:37.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:37 smithi177 ceph-mon[25767]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:37.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:37 smithi172 ceph-mon[24813]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:37.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:37 smithi172 ceph-mon[29218]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:39.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:39 smithi177 ceph-mon[25767]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:39.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:39 smithi172 ceph-mon[24813]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:39.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:39 smithi172 ceph-mon[29218]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:41.165 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:35:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:35:41.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:41 smithi177 ceph-mon[25767]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:41 smithi172 ceph-mon[24813]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:41 smithi172 ceph-mon[29218]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:42 smithi177 ceph-mon[25767]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:42 smithi172 ceph-mon[24813]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:42 smithi172 ceph-mon[29218]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:43.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:44.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:44 smithi177 ceph-mon[25767]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:44 smithi172 ceph-mon[24813]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:44 smithi172 ceph-mon[29218]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:47.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:35:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:46] "GET /metrics HTTP/1.1" 200 208016 "" "Prometheus/2.33.4" 2024-03-21T12:35:47.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:47 smithi177 ceph-mon[25767]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:47.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:47 smithi172 ceph-mon[24813]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:47.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:47 smithi172 ceph-mon[29218]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:49.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:49 smithi177 ceph-mon[25767]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:49.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:49 smithi172 ceph-mon[24813]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:49 smithi172 ceph-mon[29218]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:51.167 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:35:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:35:51.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:51 smithi172 ceph-mon[24813]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:51.448 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:51 smithi172 ceph-mon[29218]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:51 smithi177 ceph-mon[25767]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:52.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:52 smithi177 ceph-mon[25767]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:52.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:52 smithi172 ceph-mon[24813]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:52.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:52 smithi172 ceph-mon[29218]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:35:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:54 smithi177 ceph-mon[25767]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:54 smithi172 ceph-mon[24813]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:54 smithi172 ceph-mon[29218]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:57.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:35:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:35:56] "GET /metrics HTTP/1.1" 200 208016 "" "Prometheus/2.33.4" 2024-03-21T12:35:57.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:57 smithi177 ceph-mon[25767]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:57 smithi172 ceph-mon[24813]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:57 smithi172 ceph-mon[29218]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:35:59.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:35:59 smithi177 ceph-mon[25767]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:35:59 smithi172 ceph-mon[24813]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:35:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:35:59 smithi172 ceph-mon[29218]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:01.171 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:36:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:36:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:01 smithi177 ceph-mon[25767]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:01.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:01 smithi172 ceph-mon[24813]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:01.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:01 smithi172 ceph-mon[29218]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:02 smithi177 ceph-mon[25767]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:02.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:02 smithi172 ceph-mon[24813]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:02 smithi172 ceph-mon[29218]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:03.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:03 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:03 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:03 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:04.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:04 smithi177 ceph-mon[25767]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:04 smithi172 ceph-mon[24813]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:04 smithi172 ceph-mon[29218]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:07.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:36:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:06] "GET /metrics HTTP/1.1" 200 208016 "" "Prometheus/2.33.4" 2024-03-21T12:36:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:07 smithi177 ceph-mon[25767]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:07 smithi172 ceph-mon[24813]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:07 smithi172 ceph-mon[29218]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:09.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:09 smithi177 ceph-mon[25767]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:09.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:09 smithi172 ceph-mon[24813]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:09 smithi172 ceph-mon[29218]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:11.173 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:36:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:36:11.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:11 smithi177 ceph-mon[25767]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:11 smithi172 ceph-mon[24813]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:11 smithi172 ceph-mon[29218]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:12 smithi177 ceph-mon[25767]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:12 smithi172 ceph-mon[24813]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:12 smithi172 ceph-mon[29218]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:14.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:14 smithi177 ceph-mon[25767]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:14 smithi172 ceph-mon[24813]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:14 smithi172 ceph-mon[29218]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:17.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:36:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:16] "GET /metrics HTTP/1.1" 200 208027 "" "Prometheus/2.33.4" 2024-03-21T12:36:17.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:17 smithi177 ceph-mon[25767]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:17.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:17 smithi172 ceph-mon[24813]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:17 smithi172 ceph-mon[29218]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:19.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:19 smithi177 ceph-mon[25767]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:19.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:19 smithi172 ceph-mon[24813]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:19 smithi172 ceph-mon[29218]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:21.187 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:36:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:36:21.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:21 smithi172 ceph-mon[24813]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:21.448 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:21 smithi172 ceph-mon[29218]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:21 smithi177 ceph-mon[25767]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:22 smithi177 ceph-mon[25767]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:22 smithi172 ceph-mon[24813]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:22 smithi172 ceph-mon[29218]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:23.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:23 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:23 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:23 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:24 smithi177 ceph-mon[25767]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:36:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:36:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:36:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:36:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[24813]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:36:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:36:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:36:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:36:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[29218]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:36:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:36:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:36:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:36:27.138 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:36:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:26] "GET /metrics HTTP/1.1" 200 208011 "" "Prometheus/2.33.4" 2024-03-21T12:36:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[24813]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:36:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:36:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:36:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:36:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[29218]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:36:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:36:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:36:27.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:27 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:36:27.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:27 smithi177 ceph-mon[25767]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:36:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:36:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:36:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:27 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:36:29.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:29 smithi177 ceph-mon[25767]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:29.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:29 smithi172 ceph-mon[24813]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:29 smithi172 ceph-mon[29218]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:31.178 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:36:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:36:31.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:31 smithi177 ceph-mon[25767]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:31 smithi172 ceph-mon[24813]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:31 smithi172 ceph-mon[29218]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:32 smithi177 ceph-mon[25767]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:32 smithi172 ceph-mon[24813]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:32 smithi172 ceph-mon[29218]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:33.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:34.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:34 smithi177 ceph-mon[25767]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:34 smithi172 ceph-mon[24813]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:34 smithi172 ceph-mon[29218]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:37.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:36:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:36] "GET /metrics HTTP/1.1" 200 208011 "" "Prometheus/2.33.4" 2024-03-21T12:36:37.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:37 smithi177 ceph-mon[25767]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:37.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:37 smithi172 ceph-mon[24813]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:37.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:37 smithi172 ceph-mon[29218]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:39.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:39 smithi177 ceph-mon[25767]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:39.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:39 smithi172 ceph-mon[24813]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:39.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:39 smithi172 ceph-mon[29218]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:41.182 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:36:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:36:41.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:41 smithi177 ceph-mon[25767]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:41 smithi172 ceph-mon[24813]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:41 smithi172 ceph-mon[29218]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:42.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:42 smithi177 ceph-mon[25767]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:42 smithi172 ceph-mon[24813]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:42 smithi172 ceph-mon[29218]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:43.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:44 smithi177 ceph-mon[25767]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:44 smithi172 ceph-mon[24813]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:44 smithi172 ceph-mon[29218]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:47.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:36:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:46] "GET /metrics HTTP/1.1" 200 208012 "" "Prometheus/2.33.4" 2024-03-21T12:36:47.561 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:47 smithi177 ceph-mon[25767]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:47.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:47 smithi172 ceph-mon[24813]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:47.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:47 smithi172 ceph-mon[29218]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:49.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:49 smithi177 ceph-mon[25767]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:49.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:49 smithi172 ceph-mon[24813]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:49 smithi172 ceph-mon[29218]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:51.185 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:36:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:36:51.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:51 smithi172 ceph-mon[24813]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:51.448 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:51 smithi172 ceph-mon[29218]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:51 smithi177 ceph-mon[25767]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:52 smithi177 ceph-mon[25767]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:52.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:52 smithi172 ceph-mon[24813]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:52.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:52 smithi172 ceph-mon[29218]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:53.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:36:54.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:54 smithi177 ceph-mon[25767]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:54 smithi172 ceph-mon[24813]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:54 smithi172 ceph-mon[29218]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:57.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:36:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:36:56] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-03-21T12:36:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:57 smithi177 ceph-mon[25767]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:57 smithi172 ceph-mon[24813]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:57 smithi172 ceph-mon[29218]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:36:59.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:36:59 smithi177 ceph-mon[25767]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:36:59 smithi172 ceph-mon[24813]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:36:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:36:59 smithi172 ceph-mon[29218]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:01.191 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:37:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:37:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:01 smithi177 ceph-mon[25767]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:01.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:01 smithi172 ceph-mon[24813]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:01 smithi172 ceph-mon[29218]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:02 smithi177 ceph-mon[25767]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:02.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:02 smithi172 ceph-mon[24813]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:02 smithi172 ceph-mon[29218]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:03 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:03 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:03 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:04.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:04 smithi177 ceph-mon[25767]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:04 smithi172 ceph-mon[24813]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:04 smithi172 ceph-mon[29218]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:07.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:37:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:06] "GET /metrics HTTP/1.1" 200 207970 "" "Prometheus/2.33.4" 2024-03-21T12:37:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:07 smithi177 ceph-mon[25767]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:07 smithi172 ceph-mon[24813]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:07 smithi172 ceph-mon[29218]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:09.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:09 smithi177 ceph-mon[25767]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:09.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:09 smithi172 ceph-mon[24813]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:09 smithi172 ceph-mon[29218]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:11.194 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:37:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:37:11.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:11 smithi177 ceph-mon[25767]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:11 smithi172 ceph-mon[24813]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:11 smithi172 ceph-mon[29218]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:12.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:12 smithi177 ceph-mon[25767]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:12 smithi172 ceph-mon[24813]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:12 smithi172 ceph-mon[29218]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:13.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:14.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-21T12:37:14.754 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:14 smithi172 ceph-mon[24813]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:14.754 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:14 smithi172 ceph-mon[29218]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:14.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:14 smithi177 ceph-mon[25767]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:17.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:37:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:16] "GET /metrics HTTP/1.1" 200 207965 "" "Prometheus/2.33.4" 2024-03-21T12:37:17.041 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-21T12:37:17.354 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:17 smithi172 ceph-mon[24813]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:17.354 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:17 smithi172 ceph-mon[29218]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:17.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:17 smithi177 ceph-mon[25767]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:19.380 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:19 smithi172 ceph-mon[24813]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:19.380 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:19 smithi172 ceph-mon[29218]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:19.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:19 smithi177 ceph-mon[25767]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:37:19.705 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-21T12:37:21.194 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:37:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:37:21.217 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:21 smithi172 ceph-mon[24813]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:21.217 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:21 smithi172 ceph-mon[29218]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:21 smithi177 ceph-mon[25767]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:22.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:37:22.772 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:22 smithi172 ceph-mon[24813]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:22.772 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:22 smithi172 ceph-mon[29218]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:22 smithi177 ceph-mon[25767]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:23.754 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:23 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:23.755 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:23 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:23.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:23 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:24.356 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:37:24.356 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (3m) 2m ago 4m 12.9M - ba2b418f427c c3c4786e0915 2024-03-21T12:37:24.356 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (3m) 2m ago 3m 46.3M - 8.3.5 dad864ee21e9 13ce13c46ce3 2024-03-21T12:37:24.356 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (3m) 2m ago 3m 69.9M - 3.5 e1d6a67b021e 7b45bb176d18 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e ddb586f2f11e 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:9283 running (9m) 2m ago 9m 454M - 17.2.0 e1d6a67b021e 007372e25405 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (9m) 2m ago 10m 42.0M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (8m) 2m ago 8m 37.5M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (8m) 2m ago 8m 37.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (4m) 2m ago 4m 16.0M - 1dbe0e931976 1f7b7af7cf76 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (4m) 2m ago 4m 17.0M - 1dbe0e931976 7b83663275ad 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (8m) 2m ago 8m 49.9M 4016M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (7m) 2m ago 7m 50.7M 4016M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (7m) 2m ago 7m 42.8M 4016M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (6m) 2m ago 6m 44.8M 4016M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (6m) 2m ago 6m 45.2M 4272M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (6m) 2m ago 6m 43.2M 4272M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:37:24.357 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (5m) 2m ago 5m 44.7M 4272M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:37:24.358 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (5m) 2m ago 5m 46.1M 4272M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:37:24.358 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (3m) 2m ago 4m 42.2M - 514e6a882f6e 7201877bce1b 2024-03-21T12:37:24.358 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (3m) 2m ago 3m 69.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:37:24.358 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (3m) 2m ago 3m 71.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[24813]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[29218]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:24.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:37:24.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:37:24.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:37:24.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:37:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:24 smithi177 ceph-mon[25767]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:24.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:37:24.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:37:24.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:37:24.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:37:24.981 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T12:37:25.538 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:25 smithi172 ceph-mon[24813]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:25.539 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:25 smithi172 ceph-mon[29218]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:25.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:25 smithi177 ceph-mon[25767]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:26.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:26 smithi177 ceph-mon[25767]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:26.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:26 smithi172 ceph-mon[24813]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:26.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:37:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:26] "GET /metrics HTTP/1.1" 200 207957 "" "Prometheus/2.33.4" 2024-03-21T12:37:26.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:26 smithi172 ceph-mon[29218]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:26.944 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:37:26.944 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:37:26.944 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:37:26.945 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:37:27.564 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph -s' 2024-03-21T12:37:27.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:27 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3874129313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:37:27.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:37:27.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:37:27.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:27 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:37:27.894 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3874129313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:37:27.894 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:37:27.894 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:37:27.895 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:37:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3874129313' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:37:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:37:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:37:27.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:27 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:37:28.894 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:28 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:37:28.895 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:28 smithi172 ceph-mon[24813]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:28.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:28 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:37:28.895 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:28 smithi172 ceph-mon[29218]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:28 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:37:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:28 smithi177 ceph-mon[25767]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: cluster: 2024-03-21T12:37:29.637 INFO:teuthology.orchestra.run.smithi172.stdout: id: 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: health: HEALTH_OK 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: services: 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: mgr: y(active, since 5m), standbys: x 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: data: 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: pools: 6 pools, 161 pgs 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: objects: 209 objects, 585 KiB 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: pgs: 161 active+clean 2024-03-21T12:37:29.638 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:37:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: io: 2024-03-21T12:37:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T12:37:29.639 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:37:29.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:29 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2897890875' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:37:29.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:29 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2897890875' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:37:30.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:29 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2897890875' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:37:30.259 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ls' 2024-03-21T12:37:30.880 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:30 smithi172 ceph-mon[24813]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:30.881 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:30 smithi172 ceph-mon[29218]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:31.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:37:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:37:31.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:30 smithi177 ceph-mon[25767]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:32.135 INFO:teuthology.orchestra.run.smithi172.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi172=a;count:1 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:grafana ?:3000 1/1 3m ago 5m smithi177=a;count:1 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:mgr 2/2 3m ago 8m smithi172=y;smithi177=x;count:2 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:mon 3/3 3m ago 9m smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b;count:3 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi172=a;smithi177=b;count:2 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:osd 8 3m ago - 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus ?:9095 1/1 3m ago 5m smithi177=a;count:1 2024-03-21T12:37:32.136 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-03-21T12:37:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:32 smithi172 ceph-mon[24813]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:32 smithi172 ceph-mon[24813]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:32 smithi172 ceph-mon[29218]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:32 smithi172 ceph-mon[29218]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:32 smithi177 ceph-mon[25767]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:32 smithi177 ceph-mon[25767]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:32.881 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-21T12:37:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:34.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:34 smithi177 ceph-mon[25767]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:34 smithi172 ceph-mon[29218]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:34 smithi172 ceph-mon[24813]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:35.149 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled to redeploy mgr.x on host 'smithi177' 2024-03-21T12:37:35.796 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3527204077' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:37:35.796 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[24813]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:35.796 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:37:35.796 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[24813]: Schedule redeploy daemon mgr.x 2024-03-21T12:37:35.797 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:37:35.797 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3527204077' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:37:35.797 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[29218]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:35.797 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:37:35.797 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[29218]: Schedule redeploy daemon mgr.x 2024-03-21T12:37:35.797 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:35 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:37:35.820 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps --refresh' 2024-03-21T12:37:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:35 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3527204077' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:37:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:35 smithi177 ceph-mon[25767]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:35 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:37:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:35 smithi177 ceph-mon[25767]: Schedule redeploy daemon mgr.x 2024-03-21T12:37:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:35 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:37:36.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:36 smithi177 ceph-mon[25767]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:36 smithi172 ceph-mon[24813]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:36 smithi172 ceph-mon[29218]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:36.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:37:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:36] "GET /metrics HTTP/1.1" 200 207957 "" "Prometheus/2.33.4" 2024-03-21T12:37:38.072 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:37:38.072 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (3m) 3m ago 4m 12.9M - ba2b418f427c c3c4786e0915 2024-03-21T12:37:38.072 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (3m) 3m ago 3m 46.3M - 8.3.5 dad864ee21e9 13ce13c46ce3 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (3m) 3m ago 3m 69.9M - 3.5 e1d6a67b021e 7b45bb176d18 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e ddb586f2f11e 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e 007372e25405 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (10m) 3m ago 10m 42.0M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (8m) 3m ago 8m 37.5M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (9m) 3m ago 9m 37.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (4m) 3m ago 4m 16.0M - 1dbe0e931976 1f7b7af7cf76 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (4m) 3m ago 4m 17.0M - 1dbe0e931976 7b83663275ad 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (8m) 3m ago 8m 49.9M 4016M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (7m) 3m ago 7m 50.7M 4016M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (7m) 3m ago 7m 42.8M 4016M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (7m) 3m ago 7m 44.8M 4016M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (6m) 3m ago 6m 45.2M 4272M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:37:38.073 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (6m) 3m ago 6m 43.2M 4272M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:37:38.074 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (6m) 3m ago 6m 44.7M 4272M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:37:38.074 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (5m) 3m ago 5m 46.1M 4272M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:37:38.074 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (3m) 3m ago 4m 42.2M - 514e6a882f6e 7201877bce1b 2024-03-21T12:37:38.074 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (3m) 3m ago 3m 69.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:37:38.074 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (3m) 3m ago 3m 71.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:37:38.734 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 180' 2024-03-21T12:37:39.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:39 smithi172 ceph-mon[24813]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:39.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:39 smithi172 ceph-mon[24813]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:39.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:39 smithi172 ceph-mon[29218]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:39.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:39 smithi172 ceph-mon[29218]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:39.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:39 smithi177 ceph-mon[25767]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:37:39.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:39 smithi177 ceph-mon[25767]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:41.197 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:37:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:37:41.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:41 smithi177 ceph-mon[25767]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:41 smithi172 ceph-mon[24813]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:41 smithi172 ceph-mon[29218]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:42 smithi172 ceph-mon[24813]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:42.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:42 smithi172 ceph-mon[29218]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:42 smithi177 ceph-mon[25767]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:43.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:44 smithi177 ceph-mon[25767]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:44 smithi172 ceph-mon[24813]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:44 smithi172 ceph-mon[29218]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:47.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:37:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:46] "GET /metrics HTTP/1.1" 200 207967 "" "Prometheus/2.33.4" 2024-03-21T12:37:47.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:47 smithi177 ceph-mon[25767]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:47.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:47 smithi172 ceph-mon[24813]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:47.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:47 smithi172 ceph-mon[29218]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:49.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:49 smithi177 ceph-mon[25767]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:49.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:49 smithi172 ceph-mon[24813]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:49 smithi172 ceph-mon[29218]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:51.200 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:37:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:37:51.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:51 smithi177 ceph-mon[25767]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:51 smithi172 ceph-mon[24813]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:51 smithi172 ceph-mon[29218]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:52 smithi177 ceph-mon[25767]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:52.881 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:52 smithi172 ceph-mon[24813]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:52.881 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:52 smithi172 ceph-mon[29218]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:37:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:54 smithi177 ceph-mon[25767]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:54 smithi172 ceph-mon[24813]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:54 smithi172 ceph-mon[29218]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:57.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:37:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:37:56] "GET /metrics HTTP/1.1" 200 207968 "" "Prometheus/2.33.4" 2024-03-21T12:37:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:57 smithi177 ceph-mon[25767]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:57 smithi172 ceph-mon[24813]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:57 smithi172 ceph-mon[29218]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:37:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:37:59 smithi177 ceph-mon[25767]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:37:59 smithi172 ceph-mon[24813]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:37:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:37:59 smithi172 ceph-mon[29218]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:01.204 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:38:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:01 smithi177 ceph-mon[25767]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:01.645 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:01 smithi172 ceph-mon[24813]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:01 smithi172 ceph-mon[29218]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:02.764 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:02 smithi172 ceph-mon[24813]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:02.764 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:02 smithi172 ceph-mon[29218]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:02 smithi177 ceph-mon[25767]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:03.784 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:03 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:03.784 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:03 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:03.856 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:03 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:04 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:04 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:04 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:38:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:04 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:04 smithi177 ceph-mon[25767]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:38:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[24813]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:05.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:38:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:05.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:04 smithi172 ceph-mon[29218]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:07.058 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:38:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:06] "GET /metrics HTTP/1.1" 200 207968 "" "Prometheus/2.33.4" 2024-03-21T12:38:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:07 smithi172 ceph-mon[24813]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:07 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:07 smithi172 ceph-mon[29218]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:07 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:07.498 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:07 smithi177 ceph-mon[25767]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:07.498 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:07 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: Deploying daemon mgr.x on smithi177 2024-03-21T12:38:08.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:08 smithi177 ceph-mon[25767]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: Deploying daemon mgr.x on smithi177 2024-03-21T12:38:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[24813]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: Deploying daemon mgr.x on smithi177 2024-03-21T12:38:08.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:08 smithi172 ceph-mon[29218]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:11.203 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:38:11.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:11 smithi177 ceph-mon[25767]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:11 smithi172 ceph-mon[24813]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:11 smithi172 ceph-mon[29218]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:12.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:12 smithi177 ceph-mon[25767]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:12 smithi172 ceph-mon[24813]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:12 smithi172 ceph-mon[29218]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:13.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:14.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:14 smithi177 ceph-mon[25767]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:14.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:14 smithi172 ceph-mon[24813]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:14.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:14 smithi172 ceph-mon[29218]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:17.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:38:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:16] "GET /metrics HTTP/1.1" 200 207985 "" "Prometheus/2.33.4" 2024-03-21T12:38:17.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:17 smithi177 ceph-mon[25767]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:17.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:17 smithi172 ceph-mon[24813]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:17 smithi172 ceph-mon[29218]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:19.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:19 smithi177 ceph-mon[25767]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:19.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:19 smithi172 ceph-mon[24813]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:19.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:19 smithi172 ceph-mon[29218]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:21.210 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:38:21.467 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:21 smithi172 ceph-mon[24813]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:21.467 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:21 smithi172 ceph-mon[29218]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:21.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:21 smithi177 ceph-mon[25767]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:22 smithi177 ceph-mon[25767]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:22 smithi172 ceph-mon[24813]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:22 smithi172 ceph-mon[29218]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:23.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:23 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:23 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:23 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:23.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:38:23.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:23.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:24.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:24 smithi177 ceph-mon[25767]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:24.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:38:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:38:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:38:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[24813]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[29218]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:38:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:38:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:38:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:38:27.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:38:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:26] "GET /metrics HTTP/1.1" 200 207985 "" "Prometheus/2.33.4" 2024-03-21T12:38:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:27 smithi177 ceph-mon[25767]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:27.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:27 smithi172 ceph-mon[24813]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:27 smithi172 ceph-mon[29218]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:29.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:29 smithi177 ceph-mon[25767]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:29.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:29 smithi172 ceph-mon[24813]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:29 smithi172 ceph-mon[29218]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:31.212 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[26946]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:38:31.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:31 smithi177 ceph-mon[25767]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:31 smithi172 ceph-mon[24813]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:31 smithi172 ceph-mon[29218]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:32 smithi177 ceph-mon[25767]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:32 smithi172 ceph-mon[24813]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:32 smithi172 ceph-mon[29218]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:33.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:38:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:38:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:34 smithi177 ceph-mon[25767]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:34 smithi172 ceph-mon[24813]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:34.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:34 smithi172 ceph-mon[29218]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:35.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 systemd[1]: Stopping Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:38:35.373 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 bash[45825]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-21T12:38:35.659 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 podman[45836]: 2024-03-21 12:38:35.405094861 +0000 UTC m=+0.122053028 container died ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, ceph=True, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git) 2024-03-21T12:38:35.909 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 podman[45836]: 2024-03-21 12:38:35.741864866 +0000 UTC m=+0.458823034 container cleanup ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, ceph=True, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_BRANCH=HEAD, release=754, name=centos-stream) 2024-03-21T12:38:35.910 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 bash[45836]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x 2024-03-21T12:38:36.146 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:35.722Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T12:38:36.170 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 podman[45850]: 2024-03-21 12:38:35.910134453 +0000 UTC m=+0.501907524 container remove ddb586f2f11e36f5e972b72eb1f490b8658752f348500588e4d3a377730afba9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True) 2024-03-21T12:38:36.170 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:38:36.171 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 bash[45863]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr.x" found: no such container 2024-03-21T12:38:36.171 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-21T12:38:36.171 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 systemd[1]: Stopped Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:38:36.171 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:35 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service: Consumed 17.011s CPU time. 2024-03-21T12:38:36.171 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 systemd[1]: Starting Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:38:36.566 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 podman[45968]: 2024-03-21 12:38:36.287727912 +0000 UTC m=+0.017683413 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:38:36.566 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 podman[45968]: 2024-03-21 12:38:36.40328972 +0000 UTC m=+0.133245255 container create 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-21T12:38:36.832 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 podman[45968]: 2024-03-21 12:38:36.805742979 +0000 UTC m=+0.535698473 container init 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:38:36.832 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 podman[45968]: 2024-03-21 12:38:36.808438841 +0000 UTC m=+0.538394336 container start 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-21T12:38:36.999 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:38:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:36] "GET /metrics HTTP/1.1" 200 207985 "" "Prometheus/2.33.4" 2024-03-21T12:38:37.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 ceph-mgr[45987]: -- 172.21.15.177:0/3960077492 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561cb37805a0 con 0x561cb3755000 2024-03-21T12:38:37.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 bash[45968]: 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f 2024-03-21T12:38:37.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:36 smithi177 systemd[1]: Started Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:38:37.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[24813]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:37.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:37.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:37.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:38:37.299 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[29218]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:37.300 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:37 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:37.377+0000 7fcc34d5d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-mon[25767]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:38:37.475 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:38:37.796 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:37 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:37.474+0000 7fcc34d5d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:38:38.414 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:38.414 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-mon[25767]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:38.414 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:38.158+0000 7fcc34d5d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:38:38.414 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:38.254+0000 7fcc34d5d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:38:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:38 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:38 smithi172 ceph-mon[29218]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:38 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:38 smithi172 ceph-mon[24813]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:38.760 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:38.413+0000 7fcc34d5d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:38:38.761 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:38.513+0000 7fcc34d5d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:38:38.761 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:38.604+0000 7fcc34d5d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:38:39.019 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:38.890+0000 7fcc34d5d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:38:39.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:39.018+0000 7fcc34d5d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:38:39.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:39.118+0000 7fcc34d5d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:38:39.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:39.233+0000 7fcc34d5d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:38:39.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:39.416+0000 7fcc34d5d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:38:40.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:40.446+0000 7fcc34d5d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:38:40.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:40.585+0000 7fcc34d5d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:38:41.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:40.945+0000 7fcc34d5d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:38:41.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:41.036+0000 7fcc34d5d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:38:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:41 smithi177 ceph-mon[25767]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:41.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:41.238+0000 7fcc34d5d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:38:41.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:41.437+0000 7fcc34d5d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:38:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:41 smithi172 ceph-mon[24813]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:41 smithi172 ceph-mon[29218]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:42.007 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:41.741+0000 7fcc34d5d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:38:42.279 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:42.007+0000 7fcc34d5d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:38:42.279 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:42.247+0000 7fcc34d5d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:38:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:42 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:42 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:42 smithi177 ceph-mon[25767]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:42.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:42 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:42.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:42 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:42.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:42 smithi172 ceph-mon[29218]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:42.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:42 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:42.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:42 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:38:42.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:42 smithi172 ceph-mon[24813]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:43.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:43.072+0000 7fcc34d5d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:38:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:43.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:38:43.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:43.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T12:38:43.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:43.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:43.971 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:43.650+0000 7fcc34d5d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:38:43.971 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:43.749+0000 7fcc34d5d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:43.970+0000 7fcc34d5d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:38:43] ENGINE Bus STARTING 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CherryPy Checker: 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: The Application mounted at '' has an empty config. 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:38:44] ENGINE Serving on http://:::9283 2024-03-21T12:38:44.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:38:44] ENGINE Bus STARTED 2024-03-21T12:38:44.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:44.608Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:38:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: Standby manager daemon x restarted 2024-03-21T12:38:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:38:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:38:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:38:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[29218]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: Standby manager daemon x restarted 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:38:44.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:44 smithi172 ceph-mon[24813]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: Standby manager daemon x restarted 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:38:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:44 smithi177 ceph-mon[25767]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:45 smithi172 ceph-mon[24813]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-21T12:38:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:45 smithi172 ceph-mon[29218]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-21T12:38:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:45 smithi177 ceph-mon[25767]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-21T12:38:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:46 smithi172 ceph-mon[24813]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:46.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:46 smithi172 ceph-mon[29218]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:46.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:38:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:46] "GET /metrics HTTP/1.1" 200 207984 "" "Prometheus/2.33.4" 2024-03-21T12:38:47.009 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:46 smithi177 ceph-mon[25767]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:49.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:49 smithi177 ceph-mon[25767]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:49.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:49 smithi172 ceph-mon[24813]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:49 smithi172 ceph-mon[29218]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:51.217 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:38:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:38:51.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:51 smithi177 ceph-mon[25767]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:51 smithi172 ceph-mon[24813]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:51 smithi172 ceph-mon[29218]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:52.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:52 smithi177 ceph-mon[25767]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:52.865 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:52 smithi172 ceph-mon[24813]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:52.865 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:52 smithi172 ceph-mon[29218]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:38:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:38:53.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:38:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:38:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:38:53.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:38:54.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:54 smithi177 ceph-mon[25767]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:54 smithi172 ceph-mon[24813]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:54 smithi172 ceph-mon[29218]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:57.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:38:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:38:56] "GET /metrics HTTP/1.1" 200 207982 "" "Prometheus/2.33.4" 2024-03-21T12:38:57.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:57 smithi177 ceph-mon[25767]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:57 smithi172 ceph-mon[24813]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:57 smithi172 ceph-mon[29218]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:38:59.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:38:59 smithi177 ceph-mon[25767]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:38:59 smithi172 ceph-mon[24813]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:38:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:38:59 smithi172 ceph-mon[29218]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:00.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:00 smithi177 ceph-mon[25767]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:00.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:00 smithi172 ceph-mon[24813]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:00 smithi172 ceph-mon[29218]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:01.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:39:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:39:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:02 smithi177 ceph-mon[25767]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:02.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:02 smithi172 ceph-mon[24813]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:02 smithi172 ceph-mon[29218]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:03 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:03 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:03 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:03.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:39:03.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:03.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:03.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:04.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:04 smithi177 ceph-mon[25767]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:04 smithi172 ceph-mon[24813]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:04 smithi172 ceph-mon[29218]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:07.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:39:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:06] "GET /metrics HTTP/1.1" 200 207982 "" "Prometheus/2.33.4" 2024-03-21T12:39:07.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:07 smithi177 ceph-mon[25767]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:07 smithi172 ceph-mon[24813]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:07 smithi172 ceph-mon[29218]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:09.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:09 smithi177 ceph-mon[25767]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:09.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:09 smithi172 ceph-mon[24813]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:09.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:09 smithi172 ceph-mon[29218]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:10.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:10 smithi177 ceph-mon[25767]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:10 smithi172 ceph-mon[24813]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:10 smithi172 ceph-mon[29218]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:11.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:39:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:39:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:12 smithi177 ceph-mon[25767]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:12 smithi172 ceph-mon[24813]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:12 smithi172 ceph-mon[29218]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:13.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:39:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:13.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:13.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:13.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:14.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:14 smithi177 ceph-mon[25767]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:14 smithi172 ceph-mon[24813]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:14 smithi172 ceph-mon[29218]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:17.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:39:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:16] "GET /metrics HTTP/1.1" 200 207983 "" "Prometheus/2.33.4" 2024-03-21T12:39:17.560 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:17 smithi177 ceph-mon[25767]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:17.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:17 smithi172 ceph-mon[24813]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:17.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:17 smithi172 ceph-mon[29218]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:18.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:18 smithi177 ceph-mon[25767]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:18 smithi172 ceph-mon[24813]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:18 smithi172 ceph-mon[29218]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:21.226 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:39:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:39:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:21 smithi177 ceph-mon[25767]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:21.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:21 smithi172 ceph-mon[24813]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:21 smithi172 ceph-mon[29218]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:22.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:22 smithi177 ceph-mon[25767]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:22 smithi172 ceph-mon[24813]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:22 smithi172 ceph-mon[29218]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:23.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:23 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:23.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:23 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:23.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:23 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:23.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:39:23.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:23.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:23.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:23.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:24.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:39:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:39:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:39:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:39:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:24 smithi177 ceph-mon[25767]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:39:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:39:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:39:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:39:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[24813]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:39:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:39:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:39:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:39:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:24 smithi172 ceph-mon[29218]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:26.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:26 smithi177 ceph-mon[25767]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:26 smithi172 ceph-mon[24813]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:26 smithi172 ceph-mon[29218]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:27.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:39:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:26] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-03-21T12:39:28.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:28 smithi177 ceph-mon[25767]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:28 smithi172 ceph-mon[24813]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:28.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:28 smithi172 ceph-mon[29218]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:30.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:30 smithi177 ceph-mon[25767]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:30.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:30 smithi172 ceph-mon[24813]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:30 smithi172 ceph-mon[29218]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:31.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:39:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:39:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:32 smithi177 ceph-mon[25767]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:32 smithi172 ceph-mon[24813]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:32 smithi172 ceph-mon[29218]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:33.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:39:33.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:34.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:34 smithi177 ceph-mon[25767]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:34 smithi172 ceph-mon[24813]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:34 smithi172 ceph-mon[29218]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:36.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:36 smithi177 ceph-mon[25767]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:36.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:36 smithi172 ceph-mon[24813]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:36.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:36 smithi172 ceph-mon[29218]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:37.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:39:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:36] "GET /metrics HTTP/1.1" 200 207978 "" "Prometheus/2.33.4" 2024-03-21T12:39:38.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:38 smithi177 ceph-mon[25767]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:38 smithi172 ceph-mon[24813]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:38 smithi172 ceph-mon[29218]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:40.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:40 smithi177 ceph-mon[25767]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:40.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:40 smithi172 ceph-mon[24813]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:40.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:40 smithi172 ceph-mon[29218]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:41.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:39:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:39:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:42 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:39:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:42 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:39:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:42 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:39:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:42 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:39:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:42 smithi177 ceph-mon[25767]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:42.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[24813]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:39:42.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:42 smithi172 ceph-mon[29218]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:43.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:39:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:43.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:43.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:45.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:44 smithi177 ceph-mon[25767]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:44 smithi172 ceph-mon[24813]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:44 smithi172 ceph-mon[29218]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:46.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:46 smithi177 ceph-mon[25767]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:46.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:46 smithi172 ceph-mon[24813]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:46.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:46 smithi172 ceph-mon[29218]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:47.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:39:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:46] "GET /metrics HTTP/1.1" 200 207980 "" "Prometheus/2.33.4" 2024-03-21T12:39:48.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:48 smithi177 ceph-mon[25767]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:48.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:48 smithi172 ceph-mon[24813]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:48.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:48 smithi172 ceph-mon[29218]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:50.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:50 smithi177 ceph-mon[25767]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:50.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:50 smithi172 ceph-mon[24813]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:50.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:50 smithi172 ceph-mon[29218]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:51.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:39:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:39:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:52 smithi177 ceph-mon[25767]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:52.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:52 smithi172 ceph-mon[24813]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:52.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:52 smithi172 ceph-mon[29218]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:53.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:39:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:39:53.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 9 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:53.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:39:53.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:39:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:39:53.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:39:54.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:54 smithi177 ceph-mon[25767]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:54 smithi172 ceph-mon[24813]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:54 smithi172 ceph-mon[29218]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:56 smithi177 ceph-mon[25767]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:56.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:56 smithi172 ceph-mon[29218]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:56 smithi172 ceph-mon[24813]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:39:57.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:39:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:39:56] "GET /metrics HTTP/1.1" 200 207986 "" "Prometheus/2.33.4" 2024-03-21T12:39:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:39:58 smithi177 ceph-mon[25767]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:58.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:39:58 smithi172 ceph-mon[24813]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:39:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:39:58 smithi172 ceph-mon[29218]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:00.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:00 smithi177 ceph-mon[25767]: overall HEALTH_OK 2024-03-21T12:40:00.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:00 smithi172 ceph-mon[24813]: overall HEALTH_OK 2024-03-21T12:40:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:00 smithi172 ceph-mon[29218]: overall HEALTH_OK 2024-03-21T12:40:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:01 smithi177 ceph-mon[25767]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:01.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:40:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:01 smithi172 ceph-mon[24813]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:01 smithi172 ceph-mon[29218]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:02 smithi177 ceph-mon[25767]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:02.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:02 smithi172 ceph-mon[24813]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:02.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:02 smithi172 ceph-mon[29218]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:03.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:03 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:03.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:03 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:03.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:03 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:03.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:40:03.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:03.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:03.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:04 smithi177 ceph-mon[25767]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:04 smithi172 ceph-mon[24813]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:04 smithi172 ceph-mon[29218]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:06.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:06 smithi177 ceph-mon[25767]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:06 smithi172 ceph-mon[24813]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:06 smithi172 ceph-mon[29218]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:07.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:06] "GET /metrics HTTP/1.1" 200 207986 "" "Prometheus/2.33.4" 2024-03-21T12:40:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:08 smithi177 ceph-mon[25767]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:08 smithi172 ceph-mon[24813]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:08 smithi172 ceph-mon[29218]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:10.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:10 smithi177 ceph-mon[25767]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:10 smithi172 ceph-mon[24813]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:10 smithi172 ceph-mon[29218]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:11.192 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:40:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:12 smithi177 ceph-mon[25767]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:12 smithi172 ceph-mon[24813]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:12 smithi172 ceph-mon[29218]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:13.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:13.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:13.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:40:13.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:13.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:13.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:13.898 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:13.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:14.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:14 smithi177 ceph-mon[25767]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:14 smithi172 ceph-mon[24813]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:14 smithi172 ceph-mon[29218]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:16 smithi177 ceph-mon[25767]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:16.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:16 smithi172 ceph-mon[24813]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:16 smithi172 ceph-mon[29218]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:17.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:16] "GET /metrics HTTP/1.1" 200 207979 "" "Prometheus/2.33.4" 2024-03-21T12:40:18.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:18 smithi177 ceph-mon[25767]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:18 smithi172 ceph-mon[24813]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:18 smithi172 ceph-mon[29218]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:20 smithi177 ceph-mon[25767]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:20.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:20 smithi172 ceph-mon[24813]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:20.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:20 smithi172 ceph-mon[29218]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:21.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:40:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:22 smithi177 ceph-mon[25767]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:22 smithi172 ceph-mon[29218]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:22 smithi172 ceph-mon[24813]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:23.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:23 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:23 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:23 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:23.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:40:23.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:23.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:23.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:40:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:24 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:40:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:40:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:24 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:40:24.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:24 smithi177 ceph-mon[25767]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:40:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:40:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:40:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:40:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[24813]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:40:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:40:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:40:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:40:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:24 smithi172 ceph-mon[29218]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:26.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:26 smithi177 ceph-mon[25767]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:26 smithi172 ceph-mon[24813]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:26 smithi172 ceph-mon[29218]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:27.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:26] "GET /metrics HTTP/1.1" 200 207985 "" "Prometheus/2.33.4" 2024-03-21T12:40:28.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:28 smithi177 ceph-mon[25767]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:28 smithi172 ceph-mon[24813]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:28.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:28 smithi172 ceph-mon[29218]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:30.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:30 smithi177 ceph-mon[25767]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:30.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:30 smithi172 ceph-mon[24813]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:30 smithi172 ceph-mon[29218]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:31.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:40:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:32 smithi177 ceph-mon[25767]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:32.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:32 smithi172 ceph-mon[24813]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:32.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:32 smithi172 ceph-mon[29218]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:33 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:33 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:33 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:33.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:40:33.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:33.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:34.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:34 smithi177 ceph-mon[25767]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:34.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:34 smithi172 ceph-mon[24813]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:34.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:34 smithi172 ceph-mon[29218]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:36.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:36 smithi177 ceph-mon[25767]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:36 smithi172 ceph-mon[24813]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:36 smithi172 ceph-mon[29218]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:37.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:36] "GET /metrics HTTP/1.1" 200 207985 "" "Prometheus/2.33.4" 2024-03-21T12:40:38.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:38 smithi177 ceph-mon[25767]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:38 smithi172 ceph-mon[29218]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:38 smithi172 ceph-mon[24813]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:40.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:40 smithi177 ceph-mon[25767]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:40.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:40 smithi172 ceph-mon[24813]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:40.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:40 smithi172 ceph-mon[29218]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:41.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:40:42.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:42 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:40:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:42 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:40:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:42 smithi177 ceph-mon[25767]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:40:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:42 smithi177 ceph-mon[25767]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:42.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:42 smithi177 ceph-mon[25767]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:40:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:40:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:40:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[24813]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:40:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[24813]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:42.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[24813]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:40:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:40:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:40:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[29218]: from='mgr.14421 172.21.15.172:0/2899560619' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:40:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[29218]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:42.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:42 smithi172 ceph-mon[29218]: from='mgr.14421 ' entity='mgr.y' 2024-03-21T12:40:43.340 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:40:43.799 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:43 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:43.799 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:43 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:43.800 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:40:43.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:43.800 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:43.800 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:43.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:43.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:43 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:44.680 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:44 smithi172 ceph-mon[24813]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:44.680 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:44 smithi172 ceph-mon[29218]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:44.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:44 smithi177 ceph-mon[25767]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (6m) 2m ago 7m 22.6M - ba2b418f427c c3c4786e0915 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (7m) 2m ago 7m 56.0M - 8.3.5 dad864ee21e9 13ce13c46ce3 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (6m) 2m ago 6m 50.9M - 3.5 e1d6a67b021e 7b45bb176d18 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283 running (2m) 2m ago 11m 70.8M - 19.0.0-2267-g960fbbe4 19e12293512f 41407098728c 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:9283 running (13m) 2m ago 13m 461M - 17.2.0 e1d6a67b021e 007372e25405 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (13m) 2m ago 13m 45.2M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (12m) 2m ago 12m 40.3M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (12m) 2m ago 12m 38.2M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (7m) 2m ago 7m 19.2M - 1dbe0e931976 1f7b7af7cf76 2024-03-21T12:40:45.387 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (7m) 2m ago 7m 14.7M - 1dbe0e931976 7b83663275ad 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (11m) 2m ago 11m 52.2M 4016M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (11m) 2m ago 11m 52.4M 4016M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (10m) 2m ago 10m 45.5M 4016M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (10m) 2m ago 10m 48.0M 4016M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (9m) 2m ago 9m 48.6M 4272M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (9m) 2m ago 9m 47.0M 4272M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (9m) 2m ago 9m 46.7M 4272M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (8m) 2m ago 8m 49.7M 4272M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (6m) 2m ago 8m 66.7M - 514e6a882f6e 7201877bce1b 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (6m) 2m ago 6m 70.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:40:45.388 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (6m) 2m ago 6m 72.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:40:46.003 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T12:40:46.567 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:46 smithi172 ceph-mon[24813]: from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:40:46.567 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:46 smithi172 ceph-mon[24813]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:46.567 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:46 smithi172 ceph-mon[29218]: from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:40:46.568 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:46 smithi172 ceph-mon[29218]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:46 smithi177 ceph-mon[25767]: from='client.24886 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:40:46.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:46 smithi177 ceph-mon[25767]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:46.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:46] "GET /metrics HTTP/1.1" 200 207988 "" "Prometheus/2.33.4" 2024-03-21T12:40:48.013 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 1 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:40:48.014 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 1 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:40:48.015 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:40:48.327 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:48 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/330515246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:40:48.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:48 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/330515246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:40:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:48 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/330515246' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:40:48.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph -s' 2024-03-21T12:40:49.280 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:49 smithi172 ceph-mon[24813]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:49.281 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:49 smithi172 ceph-mon[29218]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:49 smithi177 ceph-mon[25767]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:50.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:50 smithi177 ceph-mon[25767]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:50.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:50 smithi172 ceph-mon[24813]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:50.639 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:50 smithi172 ceph-mon[29218]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: cluster: 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: id: 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: health: HEALTH_OK 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: services: 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: mon: 3 daemons, quorum a,c,b (age 12m) 2024-03-21T12:40:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: mgr: y(active, since 8m), standbys: x 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: data: 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: pools: 6 pools, 161 pgs 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: objects: 209 objects, 585 KiB 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: usage: 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: pgs: 161 active+clean 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: io: 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-21T12:40:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:40:51.247 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T12:40:51.291 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:40:51.541 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:51 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/382206463' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:40:51.542 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:51 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/382206463' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:40:51.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:51 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/382206463' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:40:52.820 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:52 smithi172 ceph-mon[29218]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:52.820 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:52 smithi172 ceph-mon[24813]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:52 smithi177 ceph-mon[25767]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:53.317 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_OK 2024-03-21T12:40:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:53 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:53 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/645796238' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:40:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:53.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/645796238' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:40:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:40:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/645796238' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:40:53.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:40:53.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:53.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:40:53.897 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:40:53.979 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-03-21T12:40:54.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:54 smithi177 ceph-mon[25767]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:54 smithi172 ceph-mon[24813]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:54 smithi172 ceph-mon[29218]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:56.111 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:40:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:56 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/412997873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:40:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:56 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/412997873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:40:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:56 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/412997873' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:40:56.702 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:40:56] "GET /metrics HTTP/1.1" 200 207989 "" "Prometheus/2.33.4" 2024-03-21T12:40:56.727 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph mgr fail' 2024-03-21T12:40:57.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:57 smithi172 ceph-mon[24813]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:57.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:57 smithi172 ceph-mon[29218]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:57 smithi177 ceph-mon[25767]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:40:58.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:58 smithi177 ceph-mon[25767]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:58.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:58 smithi172 ceph-mon[24813]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:58 smithi172 ceph-mon[29218]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:40:59.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/613050918' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mon[24813]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mon[24813]: Standby manager daemon y started 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:40:59.303+0000 7f29b3181700 -1 mgr handle_mgr_map I was active but no longer amignoring --setuser ceph since I am not root 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ignoring --setgroup ceph since I am not root 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mgr[25051]: -- 172.21.15.172:0/699104964 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ed7bbd0340 con 0x55ed7bba8c00 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/613050918' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:40:59.563 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mon[29218]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T12:40:59.564 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-mon[29218]: Standby manager daemon y started 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/613050918' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-mon[25767]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-mon[25767]: Standby manager daemon y started 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:40:59] ENGINE Bus STOPPING 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:40:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:40:59] ENGINE Bus STOPPED 2024-03-21T12:40:59.566 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:40:59] ENGINE Bus STARTING 2024-03-21T12:40:59.567 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:40:59] ENGINE Serving on http://:::9283 2024-03-21T12:40:59.567 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:40:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:40:59] ENGINE Bus STARTED 2024-03-21T12:40:59.821 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:40:59.629Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T12:41:00.028 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 180' 2024-03-21T12:41:00.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:40:59.821+0000 7f997d38b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:41:00.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:40:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:40:59.915+0000 7f997d38b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/613050918' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: mgrmap e22: x(active, starting, since 0.542294s), standbys: y 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:41:00.559 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: Manager daemon x is now available 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: Queued rgw.foo for migration 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:00.560 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/613050918' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: mgrmap e22: x(active, starting, since 0.542294s), standbys: y 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:41:00.561 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: Manager daemon x is now available 2024-03-21T12:41:00.562 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: Queued rgw.foo for migration 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:41:00.563 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:41:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/613050918' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:41:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: mgrmap e22: x(active, starting, since 0.542294s), standbys: y 2024-03-21T12:41:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:41:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:41:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: Manager daemon x is now available 2024-03-21T12:41:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: Queued rgw.foo for migration 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:41:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:41:00.889 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:00.648Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T12:41:00.890 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:00.557+0000 7f997d38b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:41:00.890 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:00.650+0000 7f997d38b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:41:00.890 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:00.736+0000 7f997d38b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:41:00.890 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:00.889+0000 7f997d38b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:41:01.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:00.971+0000 7f997d38b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:41:01.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:01.086+0000 7f997d38b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:41:01.317 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:00] "GET /metrics HTTP/1.1" 200 35105 "" "Prometheus/2.33.4" 2024-03-21T12:41:01.427 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: Deploying cephadm binary to smithi172 2024-03-21T12:41:01.427 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: Deploying cephadm binary to smithi177 2024-03-21T12:41:01.427 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: mgrmap e23: x(active, since 1.55278s), standbys: y 2024-03-21T12:41:01.427 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:01.427 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:01.427 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[24813]: [21/Mar/2024:12:41:00] ENGINE Bus STARTING 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:01.172+0000 7f997d38b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:01.342+0000 7f997d38b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: Deploying cephadm binary to smithi172 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: Deploying cephadm binary to smithi177 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: mgrmap e23: x(active, since 1.55278s), standbys: y 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:01.428 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:01.429 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:01.429 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-mon[29218]: [21/Mar/2024:12:41:00] ENGINE Bus STARTING 2024-03-21T12:41:01.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: Deploying cephadm binary to smithi172 2024-03-21T12:41:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: Deploying cephadm binary to smithi177 2024-03-21T12:41:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: mgrmap e23: x(active, since 1.55278s), standbys: y 2024-03-21T12:41:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:01 smithi177 ceph-mon[25767]: [21/Mar/2024:12:41:00] ENGINE Bus STARTING 2024-03-21T12:41:01.739 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:01.434+0000 7f997d38b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:41:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[24813]: [21/Mar/2024:12:41:00] ENGINE Serving on http://172.21.15.177:8765 2024-03-21T12:41:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[24813]: [21/Mar/2024:12:41:01] ENGINE Serving on https://172.21.15.177:7150 2024-03-21T12:41:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[24813]: [21/Mar/2024:12:41:01] ENGINE Bus STARTED 2024-03-21T12:41:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[24813]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[24813]: mgrmap e24: x(active, since 2s), standbys: y 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:02.484+0000 7f997d38b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:02.614+0000 7f997d38b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[29218]: [21/Mar/2024:12:41:00] ENGINE Serving on http://172.21.15.177:8765 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[29218]: [21/Mar/2024:12:41:01] ENGINE Serving on https://172.21.15.177:7150 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[29218]: [21/Mar/2024:12:41:01] ENGINE Bus STARTED 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[29218]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:02.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-mon[29218]: mgrmap e24: x(active, since 2s), standbys: y 2024-03-21T12:41:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:02 smithi177 ceph-mon[25767]: [21/Mar/2024:12:41:00] ENGINE Serving on http://172.21.15.177:8765 2024-03-21T12:41:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:02 smithi177 ceph-mon[25767]: [21/Mar/2024:12:41:01] ENGINE Serving on https://172.21.15.177:7150 2024-03-21T12:41:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:02 smithi177 ceph-mon[25767]: [21/Mar/2024:12:41:01] ENGINE Bus STARTED 2024-03-21T12:41:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:02 smithi177 ceph-mon[25767]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:02 smithi177 ceph-mon[25767]: mgrmap e24: x(active, since 2s), standbys: y 2024-03-21T12:41:03.146 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:02.823Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:41:03.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:02.809+0000 7f997d38b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:41:03.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:02.897+0000 7f997d38b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:41:03.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:03.034+0000 7f997d38b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:41:03.516 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:03.357+0000 7f997d38b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:41:03.852 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:03.596+0000 7f997d38b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:41:03.853 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:03.684+0000 7f997d38b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:41:03.853 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:03.852+0000 7f997d38b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:41:03.853 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:41:03.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:41:03.853 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:03.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T12:41:03.853 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:03.530Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:41:04.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:04.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-mon[29218]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:04.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-mon[29218]: mgrmap e25: x(active, since 4s), standbys: y 2024-03-21T12:41:04.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:04.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-mon[24813]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:04.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-mon[24813]: mgrmap e25: x(active, since 4s), standbys: y 2024-03-21T12:41:04.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:04 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:04 smithi177 ceph-mon[25767]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:04 smithi177 ceph-mon[25767]: mgrmap e25: x(active, since 4s), standbys: y 2024-03-21T12:41:05.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:04 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:04.791+0000 7f997d38b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:41:05.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:05.125+0000 7f997d38b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:41:05.387 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:05.231+0000 7f997d38b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:41:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:05.387+0000 7f997d38b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:41:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:41:05] ENGINE Bus STARTING 2024-03-21T12:41:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: CherryPy Checker: 2024-03-21T12:41:05.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: The Application mounted at '' has an empty config. 2024-03-21T12:41:05.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: 2024-03-21T12:41:05.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:41:05] ENGINE Serving on http://:::9283 2024-03-21T12:41:05.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: [21/Mar/2024:12:41:05] ENGINE Bus STARTED 2024-03-21T12:41:06.375 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:05.989Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:41:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: Standby manager daemon y restarted 2024-03-21T12:41:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: Standby manager daemon y started 2024-03-21T12:41:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:41:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: Standby manager daemon y restarted 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: Standby manager daemon y started 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:41:06.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: Standby manager daemon y restarted 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: Standby manager daemon y started 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:41:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:06 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/2180369971' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:41:07.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:41:07.595 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:07 smithi177 ceph-mon[25767]: mgrmap e26: x(active, since 7s), standbys: y 2024-03-21T12:41:07.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:07 smithi172 ceph-mon[24813]: mgrmap e26: x(active, since 7s), standbys: y 2024-03-21T12:41:07.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:07 smithi172 ceph-mon[29218]: mgrmap e26: x(active, since 7s), standbys: y 2024-03-21T12:41:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:08 smithi177 ceph-mon[25767]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:41:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:08 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:08 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:08 smithi172 ceph-mon[24813]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:41:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:08 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:08 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:08.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:08 smithi172 ceph-mon[29218]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:41:08.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:08 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:08.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:08 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:10.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi172 to 2992M 2024-03-21T12:41:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:41:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi172 to 2992M 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:11.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:41:11.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:10 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:10] "GET /metrics HTTP/1.1" 200 35105 "" "Prometheus/2.33.4" 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi172 to 2992M 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:10 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:11.860 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:11 smithi172 systemd[1]: Stopping Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:11.860 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:11 smithi172 bash[58416]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-21T12:41:11.860 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:11 smithi172 podman[58429]: 2024-03-21 12:41:11.623227864 +0000 UTC m=+0.127825978 container died 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:12.146 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:11 smithi172 podman[58429]: 2024-03-21 12:41:11.987390992 +0000 UTC m=+0.491989108 container cleanup 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:12.146 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:11 smithi172 bash[58429]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T12:41:12.146 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 bash[58455]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter.a" found: no such container 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Adjusting osd_memory_target on smithi177 to 3248M 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Adjusting osd_memory_target on smithi177 to 3248M 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:41:12.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:11 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.313 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Adjusting osd_memory_target on smithi177 to 3248M 2024-03-21T12:41:12.313 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:11 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:12.607 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:12.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.177:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.172:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:41:12.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 podman[58443]: 2024-03-21 12:41:12.297387139 +0000 UTC m=+0.676487411 container remove 1f7b7af7cf761ca9ce5ae421ad22faed3a9c289fd32e1796ec247c71495925cc (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:12.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:41:12.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T12:41:12.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 systemd[1]: Stopped Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:12.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.718s CPU time. 2024-03-21T12:41:12.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 systemd[1]: Starting Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:12.647 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 podman[58523]: 2024-03-21 12:41:12.429974566 +0000 UTC m=+0.017851975 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T12:41:12.647 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 podman[58523]: 2024-03-21 12:41:12.539982539 +0000 UTC m=+0.127859966 container create 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-mon[29218]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-21T12:41:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-mon[29218]: Reconfiguring daemon node-exporter.a on smithi172 2024-03-21T12:41:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-mon[29218]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T12:41:13.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-mon[24813]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-21T12:41:13.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-mon[24813]: Reconfiguring daemon node-exporter.a on smithi172 2024-03-21T12:41:13.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-mon[24813]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 podman[58523]: 2024-03-21 12:41:12.767126229 +0000 UTC m=+0.355003623 container init 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 podman[58523]: 2024-03-21 12:41:12.770004462 +0000 UTC m=+0.357881850 container start 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.773Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.773Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T12:41:13.147 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T12:41:13.148 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T12:41:13.149 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T12:41:13.150 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.774Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.775Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[58537]: ts=2024-03-21T12:41:12.775Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 bash[58523]: 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a 2024-03-21T12:41:13.151 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:41:12 smithi172 systemd[1]: Started Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:12 smithi177 ceph-mon[25767]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-21T12:41:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:12 smithi177 ceph-mon[25767]: Reconfiguring daemon node-exporter.a on smithi172 2024-03-21T12:41:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:12 smithi177 ceph-mon[25767]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T12:41:13.885 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=error ts=2024-03-21T12:41:13.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:41:13.885 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:13.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-21T12:41:13.885 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=warn ts=2024-03-21T12:41:13.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.177:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.177 because it doesn't contain any IP SANs" 2024-03-21T12:41:13.885 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:13.885 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:13.885 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[24813]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[24813]: Reconfiguring daemon alertmanager.a on smithi172 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[24813]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[24813]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[29218]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T12:41:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[29218]: Reconfiguring daemon alertmanager.a on smithi172 2024-03-21T12:41:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[29218]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:13 smithi172 ceph-mon[29218]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:13 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:13 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:13 smithi177 ceph-mon[25767]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T12:41:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:13 smithi177 ceph-mon[25767]: Reconfiguring daemon alertmanager.a on smithi172 2024-03-21T12:41:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:13 smithi177 ceph-mon[25767]: from='client.14781 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:13 smithi177 ceph-mon[25767]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:16.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:15 smithi177 ceph-mon[25767]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:15 smithi172 ceph-mon[24813]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:15 smithi172 ceph-mon[29218]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:16.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:41:18.680 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:18 smithi172 ceph-mon[24813]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:18.680 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:18 smithi172 ceph-mon[29218]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:18 smithi177 ceph-mon[25767]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:41:19.756 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:19 smithi172 systemd[1]: Stopping Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:19.756 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:19 smithi172 bash[58791]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T12:41:20.083 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[49259]: level=info ts=2024-03-21T12:41:19.756Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:41:20.083 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:19 smithi172 podman[58802]: 2024-03-21 12:41:19.797440195 +0000 UTC m=+0.153399241 container died c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:20.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 podman[58802]: 2024-03-21 12:41:20.173673814 +0000 UTC m=+0.529632859 container cleanup c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:20.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 bash[58802]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T12:41:20.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 bash[58831]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager.a" found: no such container 2024-03-21T12:41:20.754 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:20 smithi172 ceph-mon[24813]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:20.755 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:20 smithi172 ceph-mon[29218]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:20.755 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 podman[58816]: 2024-03-21 12:41:20.424933191 +0000 UTC m=+0.630844159 container remove c3c4786e0915aefdd1dbfde22a500ffd3e4b93ddec087092520cda1b2011920a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:20.755 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 podman[58816]: 2024-03-21 12:41:20.517419858 +0000 UTC m=+0.723330791 volume remove 6a3fed3d06f604239c247e2d57a82ed17b99463450ef20510200aebabd2861d9 2024-03-21T12:41:20.755 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T12:41:20.755 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 systemd[1]: Stopped Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:20.755 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.031s CPU time. 2024-03-21T12:41:20.755 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 systemd[1]: Starting Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:20 smithi177 ceph-mon[25767]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:21.099 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 podman[58899]: 2024-03-21 12:41:20.658058439 +0000 UTC m=+0.018395259 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-21T12:41:21.099 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 podman[58899]: 2024-03-21 12:41:20.759690285 +0000 UTC m=+0.120027094 volume create 92fa5842a4ba66a13d4983dc452870d4a108214df43c4ac25103e9d0c8002f21 2024-03-21T12:41:21.099 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:20 smithi172 podman[58899]: 2024-03-21 12:41:20.868817782 +0000 UTC m=+0.229154633 container create b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:21.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:20 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:20] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-03-21T12:41:21.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 podman[58899]: 2024-03-21 12:41:21.213479041 +0000 UTC m=+0.573815873 container init b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:21.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 podman[58899]: 2024-03-21 12:41:21.216631525 +0000 UTC m=+0.576968338 container start b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:21.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.227Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-21T12:41:21.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.228Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-21T12:41:21.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.229Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.172 port=9094 2024-03-21T12:41:21.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.230Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T12:41:21.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.256Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T12:41:21.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.257Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T12:41:21.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.259Z caller=main.go:518 msg=Listening address=:9093 2024-03-21T12:41:21.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:21.259Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-21T12:41:21.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 bash[58899]: b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c 2024-03-21T12:41:21.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:21 smithi172 systemd[1]: Started Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:22.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.177:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.172:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:22 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:41:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:22.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:22.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:41:22.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:22.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:22.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:22.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:22 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:41:23.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:23.230Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000302394s 2024-03-21T12:41:24.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-mon[25767]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:41:24.482+0000 7fcbba9b9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:24.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:24.609 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:41:24.610 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return future.result(timeout) 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.__get_result() 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: raise self._exception 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: config_blobs=daemon_spec.final_config, 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:24.611 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:24.612 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:24 smithi172 ceph-mon[24813]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:24.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:24 smithi172 ceph-mon[29218]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:25.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:25 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:25.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:41:25.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:25.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:25.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:25 smithi172 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:25 smithi172 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:25 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:25.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:25.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:41:25.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:41:25.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:25 smithi172 ceph-mon[29218]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:41:25.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:25 smithi172 ceph-mon[29218]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:41:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:25 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:41:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:25.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:41:25.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:41:25.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:25 smithi177 ceph-mon[25767]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:41:25.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:25 smithi177 ceph-mon[25767]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:41:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:26 smithi172 ceph-mon[24813]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:26 smithi172 ceph-mon[29218]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:26.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:26 smithi177 ceph-mon[25767]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:27.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:41:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[25047]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:41:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:28 smithi177 ceph-mon[25767]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:28 smithi172 ceph-mon[24813]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:28 smithi172 ceph-mon[29218]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:29.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:29 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:29.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:29 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:29.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:29 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:30.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:30 smithi177 ceph-mon[25767]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:41:30.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:30 smithi172 ceph-mon[24813]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:41:30.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:30 smithi172 ceph-mon[29218]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:41:31.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:30] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-03-21T12:41:31.646 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:41:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:41:31.233Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002983804s 2024-03-21T12:41:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:32 smithi177 ceph-mon[25767]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:41:32.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:32 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:32.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.177:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.172:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:41:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:32 smithi172 ceph-mon[24813]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:41:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:32 smithi172 ceph-mon[29218]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T12:41:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:34 smithi177 ceph-mon[25767]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:41:34.882 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:34 smithi172 ceph-mon[24813]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:41:34.882 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:34 smithi172 ceph-mon[29218]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:41:35.688 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:35.688 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 bash[49173]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:41:35.955 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:41:35.957 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T12:41:35.957 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T12:41:35.957 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T12:41:35.957 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T12:41:35.957 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[42691]: ts=2024-03-21T12:41:35.688Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T12:41:35.958 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:35 smithi177 podman[49186]: 2024-03-21 12:41:35.698379888 +0000 UTC m=+0.107709281 container died 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:36.357 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 podman[49186]: 2024-03-21 12:41:36.054731865 +0000 UTC m=+0.464061258 container cleanup 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:36.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 bash[49186]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T12:41:36.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 bash[49212]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:41:36.631 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:36 smithi177 ceph-mon[25767]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:41:36.631 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:36 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3038010341' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:41:36.632 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:36 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3701886270' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]: dispatch 2024-03-21T12:41:36.632 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:36 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]: dispatch 2024-03-21T12:41:36.632 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 podman[49200]: 2024-03-21 12:41:36.36415318 +0000 UTC m=+0.668168316 container remove 7201877bce1bd7e3237e6b649e37322f0ac0f0399e467468213097d13ca04edf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:36.632 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T12:41:36.633 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 systemd[1]: Stopped Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:36.633 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.979s CPU time. 2024-03-21T12:41:36.633 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:36.633 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 podman[49283]: 2024-03-21 12:41:36.500521911 +0000 UTC m=+0.020036733 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[29218]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3038010341' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3701886270' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]: dispatch 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]: dispatch 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[24813]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3038010341' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3701886270' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]: dispatch 2024-03-21T12:41:36.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:36 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]: dispatch 2024-03-21T12:41:37.030 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:36 smithi177 podman[49283]: 2024-03-21 12:41:36.63196359 +0000 UTC m=+0.151478443 container create caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:37 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]': finished 2024-03-21T12:41:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:37 smithi177 ceph-mon[25767]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T12:41:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:37 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/4148523404' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]: dispatch 2024-03-21T12:41:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:37 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]: dispatch 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]': finished 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[24813]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/4148523404' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]: dispatch 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]: dispatch 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1132277964"}]': finished 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[29218]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/4148523404' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]: dispatch 2024-03-21T12:41:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:37 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]: dispatch 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 podman[49283]: 2024-03-21 12:41:38.062693718 +0000 UTC m=+1.582208529 container init caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 podman[49283]: 2024-03-21 12:41:38.066474456 +0000 UTC m=+1.585989269 container start caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.094Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.095Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.095Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.095Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:41:38.316 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.095Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.095Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.096Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.097Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.099Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.102Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.102Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3µs 2024-03-21T12:41:38.317 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.102Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:41:38.582 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.316Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-21T12:41:38.582 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 bash[49283]: caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 2024-03-21T12:41:38.582 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:38.583 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.558Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-21T12:41:38.583 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.558Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-21T12:41:38.583 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.558Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=56.285µs wal_replay_duration=455.948402ms total_replay_duration=456.032715ms 2024-03-21T12:41:38.583 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.561Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:41:38.583 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.561Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T12:41:38.583 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.561Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:41:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.587Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.884034ms db_storage=763ns remote_storage=1.14µs web_handler=347ns query_engine=756ns scrape=369.832µs scrape_sd=94.245µs notify=10.1µs notify_sd=11.39µs rules=25.09917ms 2024-03-21T12:41:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:38.587Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T12:41:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]': finished 2024-03-21T12:41:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T12:41:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3787646333' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]: dispatch 2024-03-21T12:41:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]: dispatch 2024-03-21T12:41:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:38.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:38 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]': finished 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3787646333' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]: dispatch 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]: dispatch 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/623202746"}]': finished 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3787646333' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]: dispatch 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]: dispatch 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:38.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:38 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:39.357 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 systemd[1]: Stopping Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:39.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 bash[49420]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-21T12:41:39.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 podman[49431]: 2024-03-21 12:41:39.243383751 +0000 UTC m=+0.108959390 container died 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:41:39.800 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 podman[49431]: 2024-03-21 12:41:39.574611064 +0000 UTC m=+0.440186699 container cleanup 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:41:39.800 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 bash[49431]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T12:41:39.801 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:39 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]': finished 2024-03-21T12:41:39.801 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:39 smithi177 ceph-mon[25767]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T12:41:39.801 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:39 smithi177 ceph-mon[25767]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-21T12:41:39.801 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:39 smithi177 ceph-mon[25767]: Reconfiguring daemon node-exporter.b on smithi177 2024-03-21T12:41:39.801 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2529989484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/950482550"}]: dispatch 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]': finished 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[24813]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[24813]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[24813]: Reconfiguring daemon node-exporter.b on smithi177 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2529989484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/950482550"}]: dispatch 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/4144252344"}]': finished 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[29218]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[29218]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-21T12:41:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[29218]: Reconfiguring daemon node-exporter.b on smithi177 2024-03-21T12:41:39.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2529989484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/950482550"}]: dispatch 2024-03-21T12:41:40.070 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 podman[49444]: 2024-03-21 12:41:39.801192674 +0000 UTC m=+0.558292092 container remove 7b83663275adcf87f5697aa230723141c4ae8ea783881f1b51d62c2154d92e08 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:41:40.071 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:41:40.071 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 bash[49459]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter.b" found: no such container 2024-03-21T12:41:40.071 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T12:41:40.071 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 systemd[1]: Stopped Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:40.071 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.790s CPU time. 2024-03-21T12:41:40.071 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:39 smithi177 systemd[1]: Starting Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:40.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 podman[49524]: 2024-03-21 12:41:39.973554001 +0000 UTC m=+0.017484794 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-21T12:41:40.358 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 podman[49524]: 2024-03-21 12:41:40.102274376 +0000 UTC m=+0.146205197 container create 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 podman[49524]: 2024-03-21 12:41:40.312439759 +0000 UTC m=+0.356370547 container init 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 podman[49524]: 2024-03-21 12:41:40.31510174 +0000 UTC m=+0.359032522 container start 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.318Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.318Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=arp 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-21T12:41:40.359 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=edac 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-21T12:41:40.360 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=os 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-21T12:41:40.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=stat 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=time 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=uname 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.319Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-21T12:41:40.362 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[49538]: ts=2024-03-21T12:41:40.320Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-21T12:41:40.858 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 bash[49524]: 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 2024-03-21T12:41:40.858 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:41:40 smithi177 systemd[1]: Started Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2529989484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/950482550"}]': finished 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/506893588' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]: dispatch 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]: dispatch 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:40 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:41:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2529989484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/950482550"}]': finished 2024-03-21T12:41:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T12:41:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/506893588' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]: dispatch 2024-03-21T12:41:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]: dispatch 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 63 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2529989484' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/950482550"}]': finished 2024-03-21T12:41:40.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/506893588' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]: dispatch 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]: dispatch 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:40.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:40 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: Reconfiguring grafana.a (dependencies changed)... 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: Reconfiguring daemon grafana.a on smithi177 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]': finished 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1545836131' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]: dispatch 2024-03-21T12:41:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:41 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]: dispatch 2024-03-21T12:41:41.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: Reconfiguring grafana.a (dependencies changed)... 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: Reconfiguring daemon grafana.a on smithi177 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]': finished 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1545836131' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]: dispatch 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]: dispatch 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: Reconfiguring grafana.a (dependencies changed)... 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T12:41:41.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T12:41:41.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: Reconfiguring daemon grafana.a on smithi177 2024-03-21T12:41:41.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/855901005"}]': finished 2024-03-21T12:41:41.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T12:41:41.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1545836131' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]: dispatch 2024-03-21T12:41:41.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:41 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]: dispatch 2024-03-21T12:41:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[24813]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]': finished 2024-03-21T12:41:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[24813]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T12:41:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/570679114' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3184130888"}]: dispatch 2024-03-21T12:41:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[29218]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[29218]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]': finished 2024-03-21T12:41:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[29218]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T12:41:42.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:42 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/570679114' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3184130888"}]: dispatch 2024-03-21T12:41:43.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:42 smithi177 ceph-mon[25767]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:43.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:42 smithi177 ceph-mon[25767]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/3184130888"}]': finished 2024-03-21T12:41:43.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:42 smithi177 ceph-mon[25767]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T12:41:43.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:42 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/570679114' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3184130888"}]: dispatch 2024-03-21T12:41:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:43 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/570679114' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3184130888"}]': finished 2024-03-21T12:41:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:43 smithi172 ceph-mon[24813]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T12:41:43.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:43 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/570679114' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3184130888"}]': finished 2024-03-21T12:41:43.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:43 smithi172 ceph-mon[29218]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T12:41:44.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:43 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/570679114' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/3184130888"}]': finished 2024-03-21T12:41:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:43 smithi177 ceph-mon[25767]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T12:41:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:44 smithi172 ceph-mon[24813]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:44 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:44 smithi172 ceph-mon[29218]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:44 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:45.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:44 smithi177 ceph-mon[25767]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:41:45.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:44 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:46 smithi172 ceph-mon[24813]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 1 op/s 2024-03-21T12:41:46.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:46 smithi172 ceph-mon[24813]: from='client.34841 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:46.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:46 smithi172 ceph-mon[29218]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 1 op/s 2024-03-21T12:41:46.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:46 smithi172 ceph-mon[29218]: from='client.34841 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:47.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:46 smithi177 ceph-mon[25767]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 68 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 1 op/s 2024-03-21T12:41:47.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:46 smithi177 ceph-mon[25767]: from='client.34841 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:48.799 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:48 smithi177 ceph-mon[25767]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-03-21T12:41:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:48 smithi172 ceph-mon[24813]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-03-21T12:41:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:48 smithi172 ceph-mon[29218]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 760 B/s rd, 0 op/s 2024-03-21T12:41:50.496 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 systemd[1]: Stopping Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:50.496 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 bash[49825]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-21T12:41:50.796 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[41882]: t=2024-03-21T12:41:50+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-21T12:41:50.797 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 podman[49838]: 2024-03-21 12:41:50.506565071 +0000 UTC m=+0.124922138 container died 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, name=ubi8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, io.openshift.expose-services=, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.buildah.version=1.24.2) 2024-03-21T12:41:50.797 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:50 smithi177 ceph-mon[25767]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:41:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:50 smithi172 ceph-mon[24813]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:41:50.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:50 smithi172 ceph-mon[29218]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:41:51.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 podman[49838]: 2024-03-21 12:41:50.904230539 +0000 UTC m=+0.522587596 container cleanup 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, distribution-scope=public, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, name=ubi8, version=8.5, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, release=236.1648460182, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, maintainer=Paul Cuzner ) 2024-03-21T12:41:51.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 bash[49838]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a 2024-03-21T12:41:51.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:50 smithi177 bash[49868]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana.a" found: no such container 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 podman[49852]: 2024-03-21 12:41:51.188969787 +0000 UTC m=+0.677772551 container remove 13ce13c46ce30ad0ccf195b4f1fc20697e74bb834e0eea212e965610985fae0f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=Ceph Grafana Container, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762) 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Consumed 3.382s CPU time. 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 podman[49946]: 2024-03-21 12:41:51.327737889 +0000 UTC m=+0.017573816 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T12:41:51.456 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 podman[49946]: 2024-03-21 12:41:51.45662078 +0000 UTC m=+0.146456699 container create bdcd43296203bf70e117dcb2b39269edd12811790359bd38a991de4b6b0a992e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, maintainer=Paul Cuzner , vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T12:41:51.733 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 podman[49946]: 2024-03-21 12:41:51.658973541 +0000 UTC m=+0.348809461 container init bdcd43296203bf70e117dcb2b39269edd12811790359bd38a991de4b6b0a992e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.tags=base rhel8, release=236.1648460182, description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-type=git, version=8.5, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-21T12:41:51.733 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 podman[49946]: 2024-03-21 12:41:51.662435808 +0000 UTC m=+0.352271736 container start bdcd43296203bf70e117dcb2b39269edd12811790359bd38a991de4b6b0a992e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.expose-services=, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, version=8.5) 2024-03-21T12:41:51.733 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[49961]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:41:51.733 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[49961]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:41:52.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 bash[49946]: bdcd43296203bf70e117dcb2b39269edd12811790359bd38a991de4b6b0a992e 2024-03-21T12:41:52.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:41:52.040 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:51 smithi177 podman[49977]: 2024-03-21 12:41:51.743398844 +0000 UTC m=+0.038325169 container died bdcd43296203bf70e117dcb2b39269edd12811790359bd38a991de4b6b0a992e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.openshift.tags=base rhel8, version=8.5, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.openshift.expose-services=, name=ubi8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , release=236.1648460182, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-03-21T12:41:52.040 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STOPPING 2024-03-21T12:41:52.040 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:41:52.040 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STOPPED 2024-03-21T12:41:52.040 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STARTING 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:41:51] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Serving on http://:::9283 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STARTED 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STOPPING 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STOPPED 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:51] ENGINE Bus STARTING 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Serving on http://:::9283 2024-03-21T12:41:52.041 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Bus STARTED 2024-03-21T12:41:52.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Bus STOPPING 2024-03-21T12:41:52.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:41:52.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Bus STOPPED 2024-03-21T12:41:52.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Bus STARTING 2024-03-21T12:41:52.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Serving on http://:::9283 2024-03-21T12:41:52.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:41:52] ENGINE Bus STARTED 2024-03-21T12:41:52.772 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:52 smithi177 podman[49977]: 2024-03-21 12:41:52.588461756 +0000 UTC m=+0.883388081 container remove bdcd43296203bf70e117dcb2b39269edd12811790359bd38a991de4b6b0a992e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, version=8.5, description=Ceph Grafana Container, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.component=ubi8-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T12:41:52.772 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:52 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T12:41:52.772 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:41:52 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi172.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi172.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi177.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi177.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi177.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi177.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:52 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:41:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:41:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi172.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi172.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi177.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi177.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:41:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi177.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi177.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi172.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi172.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi177.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi177.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi177.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi177.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:53.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:52 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:41:53.857 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:41:55.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:54 smithi177 ceph-mon[25767]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:41:55.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:54 smithi172 ceph-mon[24813]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:41:55.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:54 smithi172 ceph-mon[29218]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:41:56.813 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:56.813 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='client.34841 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:56.814 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='client.34841 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:56.815 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:56 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='client.34841 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:41:56.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:56.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:56.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:41:56.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:56 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:41:57.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:57 smithi177 ceph-mon[25767]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:41:57.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:57 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:41:57.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:57 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:41:57.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:57 smithi177 ceph-mon[25767]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:41:57.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[24813]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:41:57.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:41:57.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:41:57.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[24813]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:41:57.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[29218]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:41:57.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:41:57.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:41:57.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:57 smithi172 ceph-mon[29218]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:41:59.110 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:41:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:41:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:41:59.559 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:41:59.191+0000 7fcbba9b9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:59.560 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:59.561 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return future.result(timeout) 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.__get_result() 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: raise self._exception 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:41:59.562 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: config_blobs=daemon_spec.final_config, 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:41:59.563 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:41:59.564 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:41:59.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-mon[25767]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-03-21T12:41:59.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:59.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:41:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:59.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:59 smithi172 ceph-mon[24813]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-03-21T12:41:59.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:59 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:59.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:41:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:41:59.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:59 smithi172 ceph-mon[29218]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-03-21T12:41:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:59 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:41:59.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:41:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:00 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:42:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:42:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:42:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:42:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:42:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:42:00.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:42:00.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:42:00.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:42:00.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:42:00.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:00 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:42:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:42:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:42:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:42:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:42:00.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:42:00.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:00 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:42:00.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:42:00.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:42:00.904 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:42:00.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:42:01.836 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:01 smithi177 ceph-mon[25767]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-03-21T12:42:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:01 smithi172 ceph-mon[24813]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-03-21T12:42:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:01 smithi172 ceph-mon[29218]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s 2024-03-21T12:42:02.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:42:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:42:01] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-21T12:42:02.888 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:02 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-03-21T12:42:02.888 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:02 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:02.888 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:02 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:42:02.889 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:02 smithi177 podman[50619]: 2024-03-21 12:42:02.751702348 +0000 UTC m=+0.017344123 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T12:42:02.889 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:02 smithi177 podman[50619]: 2024-03-21 12:42:02.888998549 +0000 UTC m=+0.154640325 container create a3c402326a1cc756e786d42ce3b5104e531b5bbeffaf5d27aca54cd247f49b00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, description=Ceph Grafana Container, io.openshift.tags=base rhel8, name=ubi8, vendor=Red Hat, Inc., release=236.1648460182, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, version=8.5, vcs-type=git) 2024-03-21T12:42:03.149 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 podman[50619]: 2024-03-21 12:42:03.083008052 +0000 UTC m=+0.348649845 container init a3c402326a1cc756e786d42ce3b5104e531b5bbeffaf5d27aca54cd247f49b00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.24.2, vendor=Red Hat, Inc., version=8.5, vcs-type=git, release=236.1648460182, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.openshift.expose-services=, distribution-scope=public, description=Ceph Grafana Container, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-21T12:42:03.149 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 podman[50619]: 2024-03-21 12:42:03.085737287 +0000 UTC m=+0.351379064 container start a3c402326a1cc756e786d42ce3b5104e531b5bbeffaf5d27aca54cd247f49b00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-type=git, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, distribution-scope=public, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., version=8.5) 2024-03-21T12:42:03.149 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50633]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:03.149 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50633]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:03.149 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 bash[50619]: a3c402326a1cc756e786d42ce3b5104e531b5bbeffaf5d27aca54cd247f49b00 2024-03-21T12:42:03.505 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:03.505 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 podman[50649]: 2024-03-21 12:42:03.156650899 +0000 UTC m=+0.037585367 container died a3c402326a1cc756e786d42ce3b5104e531b5bbeffaf5d27aca54cd247f49b00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.24.2, vendor=Red Hat, Inc., description=Ceph Grafana Container, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, io.openshift.expose-services=, distribution-scope=public, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5) 2024-03-21T12:42:03.756 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:03 smithi177 ceph-mon[25767]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:03.756 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:03.756 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 podman[50649]: 2024-03-21 12:42:03.744717312 +0000 UTC m=+0.625651791 container remove a3c402326a1cc756e786d42ce3b5104e531b5bbeffaf5d27aca54cd247f49b00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , name=ubi8, distribution-scope=public, release=236.1648460182, vcs-type=git, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8) 2024-03-21T12:42:03.756 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T12:42:03.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:03 smithi172 ceph-mon[24813]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:03.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:03 smithi172 ceph-mon[29218]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:04.107 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:03 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T12:42:05.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:05 smithi177 ceph-mon[25767]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:05.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:05 smithi172 ceph-mon[24813]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:05.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:05 smithi172 ceph-mon[29218]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:07.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:07 smithi177 ceph-mon[25767]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:07.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:07 smithi172 ceph-mon[24813]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:07.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:07 smithi172 ceph-mon[29218]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 0 op/s 2024-03-21T12:42:09.107 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:09.736 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:09 smithi172 ceph-mon[24813]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:09.737 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:09 smithi172 ceph-mon[29218]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:09.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:09 smithi177 ceph-mon[25767]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:11.848 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:11 smithi177 ceph-mon[25767]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:11.849 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:11 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3586656508' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:42:11.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:11 smithi172 ceph-mon[24813]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:11.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:11 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3586656508' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:42:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:11 smithi172 ceph-mon[29218]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:11 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3586656508' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:42:12.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:42:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:42:11] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-21T12:42:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:13 smithi177 ceph-mon[25767]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:13.859 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:13 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:13 smithi172 ceph-mon[24813]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:13 smithi172 ceph-mon[29218]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:42:14.182 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:13 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-03-21T12:42:14.182 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:13 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:14.182 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:13 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:42:14.182 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 podman[50716]: 2024-03-21 12:42:14.085749946 +0000 UTC m=+0.110931541 container create 9499edd2b4de45deb23a02195a5d7cb621aad5fc1b4f1ed034091b721acae200 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2) 2024-03-21T12:42:14.183 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 podman[50716]: 2024-03-21 12:42:13.99235405 +0000 UTC m=+0.017535646 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 podman[50716]: 2024-03-21 12:42:14.282747126 +0000 UTC m=+0.307928721 container init 9499edd2b4de45deb23a02195a5d7cb621aad5fc1b4f1ed034091b721acae200 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=Ceph Grafana Container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, vcs-type=git, version=8.5, name=ubi8, com.redhat.component=ubi8-container, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762) 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 podman[50716]: 2024-03-21 12:42:14.285510144 +0000 UTC m=+0.310691740 container start 9499edd2b4de45deb23a02195a5d7cb621aad5fc1b4f1ed034091b721acae200 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , name=ubi8, vendor=Red Hat, Inc., io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50730]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50730]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 bash[50716]: 9499edd2b4de45deb23a02195a5d7cb621aad5fc1b4f1ed034091b721acae200 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:14.602 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:14 smithi177 podman[50746]: 2024-03-21 12:42:14.39507561 +0000 UTC m=+0.072236463 container died 9499edd2b4de45deb23a02195a5d7cb621aad5fc1b4f1ed034091b721acae200 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, io.buildah.version=1.24.2, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, name=ubi8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, maintainer=Paul Cuzner , version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-21T12:42:14.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:15.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:15 smithi177 ceph-mon[25767]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:42:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:15 smithi172 ceph-mon[24813]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:42:15.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:15 smithi172 ceph-mon[29218]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:42:17.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:17 smithi177 ceph-mon[25767]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:17 smithi172 ceph-mon[24813]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:17 smithi172 ceph-mon[29218]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:18.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:18 smithi177 podman[50746]: 2024-03-21 12:42:18.061709178 +0000 UTC m=+3.738870017 container remove 9499edd2b4de45deb23a02195a5d7cb621aad5fc1b4f1ed034091b721acae200 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, architecture=x86_64, distribution-scope=public, name=ubi8, io.openshift.expose-services=, maintainer=Paul Cuzner , vendor=Red Hat, Inc., version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-03-21T12:42:18.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:18 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T12:42:18.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:18 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T12:42:19.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:19 smithi172 ceph-mon[24813]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:19.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:19 smithi172 ceph-mon[29218]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:20.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:19 smithi177 ceph-mon[25767]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:21.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:21 smithi172 ceph-mon[24813]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:21.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:21 smithi172 ceph-mon[24813]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:21.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:21 smithi172 ceph-mon[29218]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:21.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:21 smithi172 ceph-mon[29218]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:22.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:42:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:42:21] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-21T12:42:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:21 smithi177 ceph-mon[25767]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:42:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:21 smithi177 ceph-mon[25767]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:23.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:23 smithi177 ceph-mon[25767]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:23.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:23 smithi172 ceph-mon[24813]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:23 smithi172 ceph-mon[29218]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:25.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:25 smithi177 ceph-mon[25767]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:25.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:25 smithi172 ceph-mon[24813]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:25.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:25 smithi172 ceph-mon[29218]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:27.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:27 smithi172 ceph-mon[24813]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:27.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:27 smithi172 ceph-mon[29218]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:28.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:27 smithi177 ceph-mon[25767]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:28.386 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-03-21T12:42:28.387 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:28.387 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:42:28.387 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 podman[50828]: 2024-03-21 12:42:28.24795874 +0000 UTC m=+0.017536317 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T12:42:28.387 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 podman[50828]: 2024-03-21 12:42:28.387306519 +0000 UTC m=+0.156884121 container create b9bca0cace3255bdcf227c99c717d5b33f7123db1d2c4d4532519a4faef19a2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git, io.openshift.tags=base rhel8, distribution-scope=public, vendor=Red Hat, Inc., release=236.1648460182) 2024-03-21T12:42:28.659 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 podman[50828]: 2024-03-21 12:42:28.597761629 +0000 UTC m=+0.367339195 container init b9bca0cace3255bdcf227c99c717d5b33f7123db1d2c4d4532519a4faef19a2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, version=8.5, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , release=236.1648460182, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, vendor=Red Hat, Inc.) 2024-03-21T12:42:28.659 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 podman[50828]: 2024-03-21 12:42:28.600262668 +0000 UTC m=+0.369840242 container start b9bca0cace3255bdcf227c99c717d5b33f7123db1d2c4d4532519a4faef19a2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, vendor=Red Hat, Inc., architecture=x86_64, vcs-type=git, version=8.5, distribution-scope=public, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.openshift.expose-services=, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T12:42:28.659 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50843]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:28.659 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50843]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:29.017 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:29.018 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 bash[50828]: b9bca0cace3255bdcf227c99c717d5b33f7123db1d2c4d4532519a4faef19a2c 2024-03-21T12:42:29.018 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:29.018 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:28 smithi177 podman[50859]: 2024-03-21 12:42:28.730428241 +0000 UTC m=+0.096418335 container died b9bca0cace3255bdcf227c99c717d5b33f7123db1d2c4d4532519a4faef19a2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-type=git, io.buildah.version=1.24.2, maintainer=Paul Cuzner , distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-21T12:42:29.608 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:29 smithi177 podman[50859]: 2024-03-21 12:42:29.318644259 +0000 UTC m=+0.684634388 container remove b9bca0cace3255bdcf227c99c717d5b33f7123db1d2c4d4532519a4faef19a2c (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, release=236.1648460182, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, vcs-type=git, architecture=x86_64, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, maintainer=Paul Cuzner ) 2024-03-21T12:42:29.608 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:29 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T12:42:29.608 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:29 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T12:42:29.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:29 smithi172 ceph-mon[24813]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:29.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:29 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:29.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:29 smithi172 ceph-mon[29218]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:29.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:29 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:30.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:29 smithi177 ceph-mon[25767]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:29 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:31 smithi172 ceph-mon[24813]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:31.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:31 smithi172 ceph-mon[24813]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:31 smithi172 ceph-mon[29218]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:31.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:31 smithi172 ceph-mon[29218]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:32.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:42:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:42:31] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-21T12:42:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:31 smithi177 ceph-mon[25767]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:31 smithi177 ceph-mon[25767]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:33 smithi177 ceph-mon[25767]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:33.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:33 smithi172 ceph-mon[24813]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:33.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:33 smithi172 ceph-mon[29218]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:35.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:35 smithi172 ceph-mon[24813]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:35 smithi172 ceph-mon[29218]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:36.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:35 smithi177 ceph-mon[25767]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:37 smithi172 ceph-mon[24813]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:37 smithi172 ceph-mon[29218]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:38.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:37 smithi177 ceph-mon[25767]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:39.107 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:39.626 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-03-21T12:42:39.626 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:39.626 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:42:39.627 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 podman[50929]: 2024-03-21 12:42:39.492533077 +0000 UTC m=+0.018162258 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-21T12:42:39.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:39 smithi172 ceph-mon[24813]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:39.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:39 smithi172 ceph-mon[29218]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:39.953 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:39 smithi177 ceph-mon[25767]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:39.954 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 podman[50929]: 2024-03-21 12:42:39.627346253 +0000 UTC m=+0.152975448 container create 89f3ef68ec45b4c064bc77f0db3d50fc0899032561d468723dfb4fabba561001 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, vcs-type=git, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, vendor=Red Hat, Inc., io.openshift.expose-services=, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 podman[50929]: 2024-03-21 12:42:39.954587262 +0000 UTC m=+0.480216435 container init 89f3ef68ec45b4c064bc77f0db3d50fc0899032561d468723dfb4fabba561001 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, architecture=x86_64, distribution-scope=public, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762) 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 podman[50929]: 2024-03-21 12:42:39.957286664 +0000 UTC m=+0.482915827 container start 89f3ef68ec45b4c064bc77f0db3d50fc0899032561d468723dfb4fabba561001 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.openshift.expose-services=, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-type=git) 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50943]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[50943]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:40 smithi177 bash[50929]: 89f3ef68ec45b4c064bc77f0db3d50fc0899032561d468723dfb4fabba561001 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:40 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:40.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:40 smithi177 podman[50959]: 2024-03-21 12:42:40.086620722 +0000 UTC m=+0.096838684 container died 89f3ef68ec45b4c064bc77f0db3d50fc0899032561d468723dfb4fabba561001 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, architecture=x86_64, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, io.openshift.tags=base rhel8, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, maintainer=Paul Cuzner , version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762) 2024-03-21T12:42:41.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:40 smithi177 podman[50959]: 2024-03-21 12:42:40.616961342 +0000 UTC m=+0.627179276 container remove 89f3ef68ec45b4c064bc77f0db3d50fc0899032561d468723dfb4fabba561001 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, release=236.1648460182, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, maintainer=Paul Cuzner , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-21T12:42:41.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:40 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-21T12:42:41.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:40 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T12:42:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:41 smithi172 ceph-mon[24813]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:41.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:41 smithi172 ceph-mon[24813]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:41.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:41 smithi172 ceph-mon[29218]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:41.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:41 smithi172 ceph-mon[29218]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:42.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:41 smithi177 ceph-mon[25767]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:42.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:41 smithi177 ceph-mon[25767]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:42.110 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:42:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:42:41] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-21T12:42:43.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:43 smithi177 ceph-mon[25767]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:43.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:43.895 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:43 smithi172 ceph-mon[24813]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:43 smithi172 ceph-mon[29218]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:45.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:44 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:44 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:44 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:42:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:45 smithi177 ceph-mon[25767]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:45 smithi172 ceph-mon[24813]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:45 smithi172 ceph-mon[29218]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:48.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:47 smithi177 ceph-mon[25767]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:47 smithi172 ceph-mon[24813]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:47 smithi172 ceph-mon[29218]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:49.009 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:50.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:49 smithi177 ceph-mon[25767]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:49 smithi172 ceph-mon[24813]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:49 smithi172 ceph-mon[29218]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:51.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:50 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-03-21T12:42:51.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:50 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:51.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:50 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-03-21T12:42:51.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:50 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-21T12:42:51.358 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:42:50 smithi177 systemd[1]: Failed to start Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:42:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:51 smithi177 ceph-mon[25767]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:51 smithi177 ceph-mon[25767]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:42:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:42:51] "GET /metrics HTTP/1.1" 200 37972 "" "Prometheus/2.33.4" 2024-03-21T12:42:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:51 smithi172 ceph-mon[24813]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:51 smithi172 ceph-mon[24813]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:51 smithi172 ceph-mon[29218]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:51 smithi172 ceph-mon[29218]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:42:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:53 smithi177 ceph-mon[25767]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:53.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:42:53.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:53 smithi172 ceph-mon[24813]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:53.947 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:53 smithi172 ceph-mon[29218]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:56.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:55 smithi177 ceph-mon[25767]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:56.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:55 smithi172 ceph-mon[24813]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:55 smithi172 ceph-mon[29218]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:42:58.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:57 smithi177 ceph-mon[25767]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:58.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:57 smithi172 ceph-mon[24813]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:57 smithi172 ceph-mon[29218]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:42:59.107 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:42:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:42:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:00.002 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:59 smithi177 ceph-mon[25767]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:00.003 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:43:00.003 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:42:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:00.050 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:59 smithi172 ceph-mon[24813]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:00.050 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:43:00.050 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:42:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:00.050 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:59 smithi172 ceph-mon[29218]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:00.050 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:43:00.050 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:42:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:01.848 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:01 smithi177 ceph-mon[25767]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:01.848 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:01 smithi177 ceph-mon[25767]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:02.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:01 smithi172 ceph-mon[24813]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:02.021 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:01 smithi172 ceph-mon[24813]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:02.021 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:01 smithi172 ceph-mon[29218]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:02.021 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:01 smithi172 ceph-mon[29218]: from='client.25027 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:02.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:43:01] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-21T12:43:03.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:03.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:03.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:43:03.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:43:03.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:03 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:43:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:43:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:03 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:43:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T12:43:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:43:04.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:43:04.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:04 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 873 B/s rd, 0 op/s 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:43:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:04 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:43:06.826 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:06 smithi172 ceph-mon[24813]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:06.826 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:06 smithi172 ceph-mon[29218]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:06.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-mon[25767]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:43:06.947+0000 7fcbba9b9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:07.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:07.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:43:07.360 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return future.result(timeout) 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.__get_result() 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: raise self._exception 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: config_blobs=daemon_spec.final_config, 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:07.361 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:07.362 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:07.363 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:07.363 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:07.363 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:07.363 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:07.363 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:07.363 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:08 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:43:09.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:09.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:08 smithi177 ceph-mon[25767]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:08 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:43:09.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:09.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:08 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:43:09.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:43:09.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:08 smithi172 ceph-mon[24813]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:43:09.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:43:09.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:43:09.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:08 smithi172 ceph-mon[29218]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:10 smithi177 ceph-mon[25767]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:10 smithi172 ceph-mon[24813]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:10 smithi172 ceph-mon[29218]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:43:12.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:43:11] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-21T12:43:12.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:12 smithi177 ceph-mon[25767]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-21T12:43:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:12 smithi172 ceph-mon[24813]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-21T12:43:12.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:12 smithi172 ceph-mon[29218]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-03-21T12:43:13.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:13 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:14 smithi177 ceph-mon[25767]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T12:43:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:14 smithi172 ceph-mon[24813]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T12:43:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:14 smithi172 ceph-mon[29218]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-03-21T12:43:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:16 smithi177 ceph-mon[25767]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:43:17.137 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:16 smithi172 ceph-mon[24813]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:43:17.137 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:16 smithi172 ceph-mon[29218]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:43:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:18 smithi172 ceph-mon[24813]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T12:43:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:18 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3670100599' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:43:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:18 smithi172 ceph-mon[29218]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T12:43:18.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:18 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3670100599' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:43:19.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:18 smithi177 ceph-mon[25767]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T12:43:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:18 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3670100599' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:43:19.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:20 smithi177 ceph-mon[25767]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T12:43:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:20 smithi172 ceph-mon[24813]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T12:43:21.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:20 smithi172 ceph-mon[29218]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-03-21T12:43:22.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:43:21] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T12:43:22.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:22 smithi177 ceph-mon[25767]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:43:22.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:22 smithi172 ceph-mon[24813]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:43:22.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:22 smithi172 ceph-mon[29218]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-21T12:43:23.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:25.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:24 smithi177 ceph-mon[25767]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:43:25.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:24 smithi172 ceph-mon[24813]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:43:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:24 smithi172 ceph-mon[29218]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-21T12:43:27.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:26 smithi177 ceph-mon[25767]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:26 smithi172 ceph-mon[24813]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:26 smithi172 ceph-mon[29218]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:28 smithi172 ceph-mon[24813]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:28 smithi172 ceph-mon[29218]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:29.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:28 smithi177 ceph-mon[25767]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:29.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:29 smithi177 ceph-mon[25767]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:29 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:29 smithi172 ceph-mon[24813]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:29 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:29 smithi172 ceph-mon[29218]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:29 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:31.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:30 smithi177 ceph-mon[25767]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:31.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:30 smithi172 ceph-mon[24813]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:31.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:30 smithi172 ceph-mon[29218]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T12:43:32.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:43:31] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-21T12:43:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:32 smithi177 ceph-mon[25767]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:43:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:32 smithi172 ceph-mon[24813]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:43:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:32 smithi172 ceph-mon[29218]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:43:33.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:34 smithi177 ceph-mon[25767]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:35.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:34 smithi172 ceph-mon[24813]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:34 smithi172 ceph-mon[29218]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:37.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:36 smithi177 ceph-mon[25767]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:37.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:36 smithi172 ceph-mon[24813]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:36 smithi172 ceph-mon[29218]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:39.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:38 smithi177 ceph-mon[25767]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:39.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:39.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:38 smithi172 ceph-mon[24813]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:38 smithi172 ceph-mon[29218]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:40.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:39 smithi177 ceph-mon[25767]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:39 smithi172 ceph-mon[24813]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:39 smithi172 ceph-mon[29218]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:40 smithi177 ceph-mon[25767]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:41.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:40 smithi172 ceph-mon[24813]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:40 smithi172 ceph-mon[29218]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:42.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:43:41] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-21T12:43:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:42 smithi177 ceph-mon[25767]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:42 smithi172 ceph-mon[24813]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:42 smithi172 ceph-mon[29218]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:43.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:44 smithi177 ceph-mon[25767]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:44 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:44 smithi172 ceph-mon[24813]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:44 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:44 smithi172 ceph-mon[29218]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:44 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:43:47.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:46 smithi177 ceph-mon[25767]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:46 smithi172 ceph-mon[24813]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:46 smithi172 ceph-mon[29218]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:49.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:48 smithi177 ceph-mon[25767]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:49.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:48 smithi172 ceph-mon[29218]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:49.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:48 smithi172 ceph-mon[24813]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:50.009 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:49 smithi177 ceph-mon[25767]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:49 smithi172 ceph-mon[24813]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:49 smithi172 ceph-mon[29218]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:43:51.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:50 smithi177 ceph-mon[25767]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:51.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:50 smithi172 ceph-mon[24813]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:51.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:50 smithi172 ceph-mon[29218]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:43:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:43:51] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T12:43:52.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:52 smithi177 ceph-mon[25767]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:52.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:52 smithi172 ceph-mon[24813]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:52.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:52 smithi172 ceph-mon[29218]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:53.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:55.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:54 smithi177 ceph-mon[25767]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:55.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:54 smithi172 ceph-mon[24813]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:55.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:54 smithi172 ceph-mon[29218]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:56 smithi177 ceph-mon[25767]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:57.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:56 smithi172 ceph-mon[24813]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:56 smithi172 ceph-mon[29218]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:43:59.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:58 smithi177 ceph-mon[25767]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:59.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:43:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:43:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:43:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:58 smithi172 ceph-mon[24813]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:43:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:58 smithi172 ceph-mon[29218]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:00.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:59 smithi177 ceph-mon[25767]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:43:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:59 smithi172 ceph-mon[24813]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:43:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:59 smithi172 ceph-mon[29218]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:43:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:01.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:00 smithi177 ceph-mon[25767]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:00 smithi172 ceph-mon[24813]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:00 smithi172 ceph-mon[29218]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:02.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:44:01] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:44:02.493 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-21T12:44:02.809 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:02 smithi172 ceph-mon[24813]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:02.809 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:02 smithi172 ceph-mon[29218]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:02.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:02 smithi177 ceph-mon[25767]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:03.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:05.051 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled to redeploy mgr.y on host 'smithi172' 2024-03-21T12:44:05.052 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:04 smithi172 ceph-mon[24813]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:05.052 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:04 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2639981387' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:44:05.052 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:04 smithi172 ceph-mon[29218]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:05.052 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:04 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2639981387' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:44:05.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:04 smithi177 ceph-mon[25767]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:05.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:04 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2639981387' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T12:44:05.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps --refresh' 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: Schedule redeploy daemon mgr.y 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:06.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[24813]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: Schedule redeploy daemon mgr.y 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:06.279 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:06 smithi172 ceph-mon[29218]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:06.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:44:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: Schedule redeploy daemon mgr.y 2024-03-21T12:44:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:06 smithi177 ceph-mon[25767]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:44:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (2m) 64s ago 10m 17.4M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:44:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 64s ago 10m - - 2024-03-21T12:44:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (118s) 64s ago 9m 50.3M - 3.5 e1d6a67b021e af19b2c27987 2024-03-21T12:44:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283 running (5m) 64s ago 15m 511M - 19.0.0-2267-g960fbbe4 19e12293512f 41407098728c 2024-03-21T12:44:07.852 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:9283 running (16m) 64s ago 16m 414M - 17.2.0 e1d6a67b021e 007372e25405 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (16m) 64s ago 16m 55.2M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (15m) 64s ago 15m 45.9M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (15m) 64s ago 15m 47.3M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (2m) 64s ago 11m 17.3M - 1.3.1 1dbe0e931976 1145f3da8c8f 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (2m) 64s ago 11m 18.9M - 1.3.1 1dbe0e931976 4437b302e23d 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (14m) 64s ago 14m 54.5M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (14m) 64s ago 14m 54.8M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (14m) 64s ago 14m 47.8M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (13m) 64s ago 13m 50.4M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (13m) 64s ago 13m 50.8M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (12m) 64s ago 12m 49.2M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (12m) 64s ago 12m 49.2M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (12m) 64s ago 12m 52.0M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (2m) 64s ago 11m 68.4M - 2.33.4 514e6a882f6e caf84b253e4b 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (9m) 64s ago 9m 72.2M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:44:07.853 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (9m) 64s ago 9m 74.3M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:44:08.955 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:08 smithi172 ceph-mon[24813]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:08.955 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:08 smithi172 ceph-mon[24813]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:44:08.955 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:08 smithi172 ceph-mon[29218]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:08.955 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:08 smithi172 ceph-mon[29218]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:44:09.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:08 smithi177 ceph-mon[25767]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:08 smithi177 ceph-mon[25767]: from='client.25078 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:44:09.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:09.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 180' 2024-03-21T12:44:09.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:09 smithi177 ceph-mon[25767]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:09.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:09 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:09.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:09 smithi172 ceph-mon[24813]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:09.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:09 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:09 smithi172 ceph-mon[29218]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:09.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:09 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:10 smithi172 ceph-mon[24813]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:10.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:10 smithi172 ceph-mon[29218]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:11.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:10 smithi177 ceph-mon[25767]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:12.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:44:11] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:44:12.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:12 smithi177 ceph-mon[25767]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:12 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:12 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:12.884 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:12 smithi172 ceph-mon[24813]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:12.884 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:12 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:12.884 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:12 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:12.884 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:12 smithi172 ceph-mon[29218]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:12.884 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:12 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:12.884 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:12 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:13.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:13 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:14 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:15.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:15.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:15.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:14 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[24813]: Deploying daemon mgr.y on smithi172 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:15.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:15 smithi172 ceph-mon[29218]: Deploying daemon mgr.y on smithi172 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:15 smithi177 ceph-mon[25767]: Deploying daemon mgr.y on smithi172 2024-03-21T12:44:16.773 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:16 smithi172 systemd[1]: Stopping Ceph mgr.y for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:44:17.091 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:16 smithi172 podman[62992]: 2024-03-21 12:44:16.772110012 +0000 UTC m=+0.149367034 container died 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream) 2024-03-21T12:44:17.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 podman[62992]: 2024-03-21 12:44:17.190975775 +0000 UTC m=+0.568232789 container cleanup 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, name=centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.expose-services=) 2024-03-21T12:44:17.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 bash[62992]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y 2024-03-21T12:44:17.672 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:17 smithi172 ceph-mon[24813]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-21T12:44:17.672 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:17 smithi172 ceph-mon[29218]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-21T12:44:17.673 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 podman[63006]: 2024-03-21 12:44:17.412612256 +0000 UTC m=+0.643577988 container remove 007372e254058f80c3dace563f083bcd3077df0d0e1175dc7283d3c7359f48ea (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-type=git, io.buildah.version=1.19.8, version=8, CEPH_POINT_RELEASE=-17.2.0) 2024-03-21T12:44:17.673 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:44:17.673 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-21T12:44:17.673 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 systemd[1]: Stopped Ceph mgr.y for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:44:17.673 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service: Consumed 59.670s CPU time. 2024-03-21T12:44:17.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:17 smithi177 ceph-mon[25767]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-21T12:44:18.071 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:17 smithi172 systemd[1]: Starting Ceph mgr.y for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:44:18.324 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 podman[63138]: 2024-03-21 12:44:17.974818388 +0000 UTC m=+0.018520466 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:44:18.324 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 podman[63138]: 2024-03-21 12:44:18.102414675 +0000 UTC m=+0.146116790 container create df80529fec67903ea1c3a261037832a943d5f31c81cf90231023380b6ddc6d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-21T12:44:18.324 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 podman[63138]: 2024-03-21 12:44:18.321707738 +0000 UTC m=+0.365409807 container init df80529fec67903ea1c3a261037832a943d5f31c81cf90231023380b6ddc6d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-21T12:44:18.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 podman[63138]: 2024-03-21 12:44:18.32459023 +0000 UTC m=+0.368292299 container start df80529fec67903ea1c3a261037832a943d5f31c81cf90231023380b6ddc6d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD) 2024-03-21T12:44:18.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 ceph-mgr[63157]: -- 172.21.15.172:0/805725097 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d3ac4e5a0 con 0x557d3ac22c00 2024-03-21T12:44:18.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 bash[63138]: df80529fec67903ea1c3a261037832a943d5f31c81cf90231023380b6ddc6d8e 2024-03-21T12:44:18.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 systemd[1]: Started Ceph mgr.y for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:44:19.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:19.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:18 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:18.902+0000 7f48c2a7f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:44:19.163 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:19.000+0000 7f48c2a7f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:44:19.777 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:19.684+0000 7f48c2a7f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:44:19.777 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:44:19.778 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[24813]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:44:19.779 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-mon[29218]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='client.15153 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:44:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:19 smithi177 ceph-mon[25767]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 73 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 81 op/s 2024-03-21T12:44:20.109 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:19.777+0000 7f48c2a7f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:44:20.109 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:19.924+0000 7f48c2a7f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:44:20.109 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.021+0000 7f48c2a7f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:44:20.109 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.108+0000 7f48c2a7f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:44:20.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.386+0000 7f48c2a7f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:44:20.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.501+0000 7f48c2a7f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:44:20.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.588+0000 7f48c2a7f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:44:21.008 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.686+0000 7f48c2a7f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:44:21.008 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:20.869+0000 7f48c2a7f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:44:21.804 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-mon[25767]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:44:21.798+0000 7fcbba9b9700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:44:21.805 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:44:21.806 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return future.result(timeout) 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return self.__get_result() 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: raise self._exception 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: config_blobs=daemon_spec.final_config, 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:44:21.807 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Creating ceph-iscsi config... 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: Traceback (most recent call last): 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: return _run_code(code, main_globals, None, 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: exec(code, run_globals) 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:21.808 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:21.809 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:21.809 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:21.809 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:21.809 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:21.809 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:21.809 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:21.918 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:44:21] "GET /metrics HTTP/1.1" 200 37982 "" "Prometheus/2.33.4" 2024-03-21T12:44:22.027 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:21 smithi172 ceph-mon[24813]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-21T12:44:22.027 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:21.889+0000 7f48c2a7f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:44:22.028 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:21 smithi172 ceph-mon[29218]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-21T12:44:22.395 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:22.027+0000 7f48c2a7f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:44:22.395 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:22.394+0000 7f48c2a7f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:44:22.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:22.489+0000 7f48c2a7f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:22 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:44:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:44:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:44:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:22 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:23.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:22.694+0000 7f48c2a7f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:44:23.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:22.888+0000 7f48c2a7f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:44:23.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:23.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:23.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:44:23.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:44:23.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:23.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:44:23.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:44:23.154 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:22 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:23.470 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:23.193+0000 7f48c2a7f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:44:23.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:23 smithi177 ceph-mon[25767]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-21T12:44:23.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:23.895 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:23 smithi172 ceph-mon[24813]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-21T12:44:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:23 smithi172 ceph-mon[29218]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-21T12:44:23.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:23.469+0000 7f48c2a7f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:44:23.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:23.717+0000 7f48c2a7f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:44:24.886 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:24.605+0000 7f48c2a7f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:44:25.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:25.212+0000 7f48c2a7f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:44:25.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:25.327+0000 7f48c2a7f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:44:25.660 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:25.594+0000 7f48c2a7f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:44:25.660 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:44:25] ENGINE Bus STARTING 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: Standby manager daemon y restarted 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: Standby manager daemon y started 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:25 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CherryPy Checker: 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: The Application mounted at '' has an empty config. 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:44:25] ENGINE Serving on http://:::9283 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:44:25] ENGINE Bus STARTED 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: Standby manager daemon y restarted 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: Standby manager daemon y started 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-21T12:44:26.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: Standby manager daemon y restarted 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: Standby manager daemon y started 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:26.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:25 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: mgrmap e27: x(active, since 3m), standbys: y 2024-03-21T12:44:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:26 smithi177 ceph-mon[25767]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: mgrmap e27: x(active, since 3m), standbys: y 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[24813]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 0 B/s wr, 130 op/s 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: mgrmap e27: x(active, since 3m), standbys: y 2024-03-21T12:44:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:26 smithi172 ceph-mon[29218]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:44:28.829 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[24813]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-21T12:44:28.829 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:28.829 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:28.829 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:44:28.829 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[29218]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:44:28.830 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:28 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:29.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-mon[25767]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 0 B/s wr, 79 op/s 2024-03-21T12:44:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:44:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:44:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:28 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:44:30.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:44:30.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:44:30.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/699618010' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:44:30.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:30.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:44:30.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:44:30.010 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/699618010' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:44:30.010 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:29 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:29 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T12:44:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:29 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T12:44:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:29 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/699618010' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:44:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:29 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:30.992 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:30 smithi172 ceph-mon[24813]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2024-03-21T12:44:30.993 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:30 smithi172 ceph-mon[29218]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2024-03-21T12:44:31.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:30 smithi177 ceph-mon[25767]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 113 B/s rd, 0 op/s 2024-03-21T12:44:32.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:44:31] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T12:44:32.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:32 smithi177 ceph-mon[25767]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 339 B/s rd, 0 op/s 2024-03-21T12:44:32.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:32 smithi172 ceph-mon[24813]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 339 B/s rd, 0 op/s 2024-03-21T12:44:32.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:32 smithi172 ceph-mon[29218]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 339 B/s rd, 0 op/s 2024-03-21T12:44:33.819 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:34.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 373 B/s rd, 0 op/s 2024-03-21T12:44:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 285 B/s rd, 0 op/s 2024-03-21T12:44:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:33 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 373 B/s rd, 0 op/s 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 285 B/s rd, 0 op/s 2024-03-21T12:44:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 373 B/s rd, 0 op/s 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 285 B/s rd, 0 op/s 2024-03-21T12:44:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:33 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:44:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:34 smithi177 ceph-mon[25767]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:44:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:34 smithi172 ceph-mon[24813]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:44:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:34 smithi172 ceph-mon[29218]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:44:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:35 smithi177 ceph-mon[25767]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-03-21T12:44:36.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:35 smithi172 ceph-mon[24813]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-03-21T12:44:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:35 smithi172 ceph-mon[29218]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 223 B/s rd, 0 op/s 2024-03-21T12:44:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:37 smithi177 ceph-mon[25767]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:44:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:37 smithi172 ceph-mon[24813]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:44:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:37 smithi172 ceph-mon[29218]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:44:39.107 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:39 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:39 smithi172 ceph-mon[24813]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:44:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:39 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:39 smithi172 ceph-mon[29218]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:44:40.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:39 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:39 smithi177 ceph-mon[25767]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T12:44:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:41 smithi177 ceph-mon[25767]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:44:42.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:44:41] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T12:44:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:41 smithi172 ceph-mon[24813]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:44:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:41 smithi172 ceph-mon[29218]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:44:43.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:42 smithi177 ceph-mon[25767]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:44:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:42 smithi172 ceph-mon[24813]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:44:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:42 smithi172 ceph-mon[29218]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:44:43.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:44 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:44 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:44 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:44:45.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:45 smithi177 ceph-mon[25767]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:45.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:45 smithi172 ceph-mon[24813]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:45.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:45 smithi172 ceph-mon[29218]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:47 smithi177 ceph-mon[25767]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:47.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:47 smithi172 ceph-mon[24813]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:47.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:47 smithi172 ceph-mon[29218]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:49.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:49 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:49 smithi172 ceph-mon[24813]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:49 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:49 smithi172 ceph-mon[29218]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:49 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:44:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:49 smithi177 ceph-mon[25767]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:51 smithi177 ceph-mon[25767]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:44:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:44:51] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:44:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:51 smithi172 ceph-mon[24813]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:51 smithi172 ceph-mon[29218]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:53.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:44:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:53 smithi172 ceph-mon[24813]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:53 smithi172 ceph-mon[29218]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:53 smithi177 ceph-mon[25767]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:55.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:54 smithi177 ceph-mon[25767]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:55.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:54 smithi172 ceph-mon[24813]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:54 smithi172 ceph-mon[29218]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:44:57.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:57 smithi177 ceph-mon[25767]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:57.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:57 smithi172 ceph-mon[24813]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:57.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:57 smithi172 ceph-mon[29218]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:44:59.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:44:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:44:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:59 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:59 smithi172 ceph-mon[24813]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:44:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:59 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:59 smithi172 ceph-mon[29218]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:44:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:59 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:59 smithi177 ceph-mon[25767]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:44:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:01 smithi177 ceph-mon[25767]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:02.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:45:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:45:01] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:45:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:01 smithi172 ceph-mon[24813]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:01 smithi172 ceph-mon[29218]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:03.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:02 smithi177 ceph-mon[25767]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:03.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:02 smithi172 ceph-mon[24813]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:02 smithi172 ceph-mon[29218]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:03.607 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:05 smithi172 ceph-mon[24813]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:05 smithi172 ceph-mon[29218]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:06.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:05 smithi177 ceph-mon[25767]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:07.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:07 smithi177 ceph-mon[25767]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:07.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:07 smithi172 ceph-mon[24813]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:07.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:07 smithi172 ceph-mon[29218]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:09.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:09 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:09 smithi177 ceph-mon[25767]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:10.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:09 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:10.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:09 smithi172 ceph-mon[24813]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:09 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:09 smithi172 ceph-mon[29218]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:10 smithi177 ceph-mon[25767]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:10 smithi172 ceph-mon[24813]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:11.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:10 smithi172 ceph-mon[29218]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:12.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:45:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:45:11] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:45:13.857 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:13 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:13 smithi177 ceph-mon[25767]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:13 smithi172 ceph-mon[24813]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:13 smithi172 ceph-mon[29218]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:15.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:15 smithi177 ceph-mon[25767]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:15 smithi172 ceph-mon[24813]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:15 smithi172 ceph-mon[29218]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:17.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:17 smithi177 ceph-mon[25767]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:17 smithi172 ceph-mon[24813]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:17 smithi172 ceph-mon[29218]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:19.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:19 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:19 smithi177 ceph-mon[25767]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:20.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:19 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:20.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:19 smithi172 ceph-mon[24813]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:19 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:19 smithi172 ceph-mon[29218]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:21.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:20 smithi177 ceph-mon[25767]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:20 smithi172 ceph-mon[24813]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:20 smithi172 ceph-mon[29218]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:22.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:45:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:45:21] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-21T12:45:23.857 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:24.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:23 smithi177 ceph-mon[25767]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:23 smithi172 ceph-mon[24813]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:24.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:23 smithi172 ceph-mon[29218]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:25.297 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:24 smithi172 ceph-mon[24813]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:25.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:24 smithi172 ceph-mon[29218]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:24 smithi177 ceph-mon[25767]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:27.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:27 smithi172 ceph-mon[24813]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:27.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:27 smithi172 ceph-mon[29218]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:28.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:27 smithi177 ceph-mon[25767]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:28.951 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:28 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:28 smithi177 ceph-mon[25767]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:28 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:28 smithi172 ceph-mon[24813]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:28 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:28 smithi172 ceph-mon[29218]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:30.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:29 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:30.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:29 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:30.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:29 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:30 smithi177 ceph-mon[25767]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:30 smithi172 ceph-mon[24813]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:30 smithi172 ceph-mon[29218]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:32.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:45:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:45:31] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-21T12:45:33.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:32 smithi177 ceph-mon[25767]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:32 smithi172 ceph-mon[24813]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:32 smithi172 ceph-mon[29218]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:33.857 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:45:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:45:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:45:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-mon[25767]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-21T12:45:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-mon[25767]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:33 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[29218]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[29218]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:45:34.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:45:34.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:45:34.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:45:34.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[24813]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-21T12:45:34.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[24813]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:34.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:33 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:45:37.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:36 smithi177 ceph-mon[25767]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:37.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:36 smithi172 ceph-mon[24813]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:36 smithi172 ceph-mon[29218]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:38 smithi172 ceph-mon[24813]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:38 smithi172 ceph-mon[29218]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:39.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:38 smithi177 ceph-mon[25767]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:39.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:39 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:39 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:39 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:40 smithi177 ceph-mon[25767]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-21T12:45:41.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:40 smithi172 ceph-mon[24813]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-21T12:45:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:40 smithi172 ceph-mon[29218]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-21T12:45:42.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:45:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:45:41] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-21T12:45:43.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:42 smithi177 ceph-mon[25767]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:43.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:42 smithi172 ceph-mon[24813]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:42 smithi172 ceph-mon[29218]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:45:43.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:44 smithi177 ceph-mon[25767]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:45:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:44 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:44 smithi172 ceph-mon[24813]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:45:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:44 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:44 smithi172 ceph-mon[29218]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:45:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:44 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:45:47.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:46 smithi177 ceph-mon[25767]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:46 smithi172 ceph-mon[24813]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:46 smithi172 ceph-mon[29218]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:48.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:48 smithi172 ceph-mon[24813]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:48 smithi172 ceph-mon[29218]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:49.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:48 smithi177 ceph-mon[25767]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:49.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:50.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:49 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:49 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:49 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:45:51.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:50 smithi177 ceph-mon[25767]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:51.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:50 smithi172 ceph-mon[24813]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:51.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:50 smithi172 ceph-mon[29218]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:45:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:45:51] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:45:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:52 smithi177 ceph-mon[25767]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:52 smithi172 ceph-mon[24813]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:53.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:52 smithi172 ceph-mon[29218]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:53.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:45:55.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:54 smithi177 ceph-mon[25767]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:55.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:54 smithi172 ceph-mon[24813]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:55.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:54 smithi172 ceph-mon[29218]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:56.948 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:56 smithi172 ceph-mon[24813]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:56.949 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:56 smithi172 ceph-mon[29218]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:57.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:56 smithi177 ceph-mon[25767]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:45:59.103 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:58 smithi172 ceph-mon[24813]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:59.103 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:58 smithi172 ceph-mon[29218]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:59.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:58 smithi177 ceph-mon[25767]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:45:59.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:45:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:45:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:59 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:45:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:59 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:45:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:59 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:45:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:00 smithi177 ceph-mon[25767]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:00 smithi172 ceph-mon[24813]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:00 smithi172 ceph-mon[29218]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:02.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:46:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:46:01] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:46:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:02 smithi177 ceph-mon[25767]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:03.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:02 smithi172 ceph-mon[29218]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:02 smithi172 ceph-mon[24813]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:03.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:46:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:46:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:05.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:04 smithi177 ceph-mon[25767]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:04 smithi172 ceph-mon[24813]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:05.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:04 smithi172 ceph-mon[29218]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:07.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:06 smithi177 ceph-mon[25767]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:06 smithi172 ceph-mon[24813]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:07.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:06 smithi172 ceph-mon[29218]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:08.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:08 smithi172 ceph-mon[24813]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:08.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:08 smithi172 ceph-mon[29218]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:08 smithi177 ceph-mon[25767]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:09.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:46:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:46:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:09 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:09 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:09 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:10 smithi177 ceph-mon[25767]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:10 smithi172 ceph-mon[24813]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:10 smithi172 ceph-mon[29218]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:12.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:46:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:46:11] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-21T12:46:13.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:12 smithi177 ceph-mon[25767]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:13.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:12 smithi172 ceph-mon[24813]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:12 smithi172 ceph-mon[29218]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:13.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:46:13 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:46:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:14 smithi177 ceph-mon[25767]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:14 smithi172 ceph-mon[24813]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:14 smithi172 ceph-mon[29218]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:16 smithi177 ceph-mon[25767]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:16 smithi172 ceph-mon[24813]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:17.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:16 smithi172 ceph-mon[29218]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:18.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:18 smithi172 ceph-mon[24813]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:18.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:18 smithi172 ceph-mon[29218]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:19.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:18 smithi177 ceph-mon[25767]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:19.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:46:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:46:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:19 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:20.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:19 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:19 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:20 smithi177 ceph-mon[25767]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:20 smithi172 ceph-mon[24813]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:21.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:20 smithi172 ceph-mon[29218]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:22.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:46:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:46:21] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T12:46:23.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:22 smithi177 ceph-mon[25767]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:23.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:22 smithi172 ceph-mon[24813]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:23.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:22 smithi172 ceph-mon[29218]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:23.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:46:23 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:46:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi172\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", domainname=\"(none)\", instance=\"smithi172\", job=\"node\", machine=\"x86_64\", nodename=\"smithi172\", release=\"5.14.0-430.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:24 smithi177 ceph-mon[25767]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:25.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:24 smithi172 ceph-mon[24813]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:24 smithi172 ceph-mon[29218]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:26 smithi177 ceph-mon[25767]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:26 smithi172 ceph-mon[24813]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:26 smithi172 ceph-mon[29218]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:28 smithi172 ceph-mon[24813]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:28.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:28 smithi172 ceph-mon[29218]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:28 smithi177 ceph-mon[25767]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:29.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:46:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:46:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"172.21.15.177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3bbfe794-e77e-11ee-95cd-87774f69a715\", cluster_addr=\"172.21.15.172\", device_class=\"ssd\", hostname=\"smithi172\", instance=\"smithi177:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.172\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-21T12:46:30.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:29 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:29 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:29 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:29 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:29 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:29 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:31.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:30 smithi177 ceph-mon[25767]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:31.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:30 smithi172 ceph-mon[24813]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:31.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:30 smithi172 ceph-mon[29218]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:32.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:46:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:46:31] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-21T12:46:33.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:32 smithi177 ceph-mon[25767]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:32 smithi172 ceph-mon[24813]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:32 smithi172 ceph-mon[29218]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:34.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:33 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:46:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:33 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:46:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:33 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:46:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:34 smithi177 ceph-mon[25767]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:34 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:46:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:34 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:46:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:34 smithi177 ceph-mon[25767]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[24813]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[24813]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[29218]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:46:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:34 smithi172 ceph-mon[29218]: from='mgr.34772 ' entity='mgr.x' 2024-03-21T12:46:36.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:35 smithi177 ceph-mon[25767]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:46:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:35 smithi177 ceph-mon[25767]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:46:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:35 smithi172 ceph-mon[24813]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:46:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:35 smithi172 ceph-mon[24813]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:46:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:35 smithi172 ceph-mon[29218]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:46:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:35 smithi172 ceph-mon[29218]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:46:38.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:37 smithi177 ceph-mon[25767]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:38.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:37 smithi172 ceph-mon[24813]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:37 smithi172 ceph-mon[29218]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:40.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:39 smithi177 ceph-mon[25767]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:39 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:39 smithi172 ceph-mon[24813]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:39 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:39 smithi172 ceph-mon[29218]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:39 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:42.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:46:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:46:41] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-21T12:46:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:41 smithi177 ceph-mon[25767]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:46:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:41 smithi172 ceph-mon[24813]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:46:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:41 smithi172 ceph-mon[29218]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:46:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:43 smithi177 ceph-mon[25767]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:44.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:43 smithi172 ceph-mon[24813]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:43 smithi172 ceph-mon[29218]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:46:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:44 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:44 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:45.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:44 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:46:46.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:45 smithi177 ceph-mon[25767]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:46:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:45 smithi172 ceph-mon[24813]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:46:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:45 smithi172 ceph-mon[29218]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:46:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:47 smithi177 ceph-mon[25767]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:47 smithi172 ceph-mon[24813]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:47 smithi172 ceph-mon[29218]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:50.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:49 smithi177 ceph-mon[25767]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:50.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:49 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:49 smithi172 ceph-mon[24813]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:49 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:49 smithi172 ceph-mon[29218]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:49 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:46:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:51 smithi177 ceph-mon[25767]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:46:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:46:51] "GET /metrics HTTP/1.1" 200 37980 "" "Prometheus/2.33.4" 2024-03-21T12:46:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:51 smithi172 ceph-mon[24813]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:51 smithi172 ceph-mon[29218]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:53 smithi177 ceph-mon[25767]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:53 smithi172 ceph-mon[24813]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:53 smithi172 ceph-mon[29218]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:56.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:55 smithi177 ceph-mon[25767]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:56.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:55 smithi172 ceph-mon[24813]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:55 smithi172 ceph-mon[29218]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:46:58.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:57 smithi177 ceph-mon[25767]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:58.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:57 smithi172 ceph-mon[24813]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:46:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:57 smithi172 ceph-mon[29218]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:59 smithi177 ceph-mon[25767]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:59 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:46:59 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:59 smithi172 ceph-mon[24813]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:59 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:46:59 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:59 smithi172 ceph-mon[29218]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:59 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:46:59 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:02.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:47:01] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T12:47:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:01 smithi177 ceph-mon[25767]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:01 smithi172 ceph-mon[24813]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:01 smithi172 ceph-mon[29218]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:03 smithi177 ceph-mon[25767]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:03 smithi172 ceph-mon[24813]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:03 smithi172 ceph-mon[29218]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:05 smithi177 ceph-mon[25767]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:05 smithi172 ceph-mon[24813]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:05 smithi172 ceph-mon[29218]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:08.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:07 smithi177 ceph-mon[25767]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:07 smithi172 ceph-mon[24813]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:07 smithi172 ceph-mon[29218]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:09 smithi177 ceph-mon[25767]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:09 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:09 smithi172 ceph-mon[24813]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:09 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:09 smithi172 ceph-mon[29218]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:09 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:11.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:11 smithi172 ceph-mon[24813]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:11 smithi172 ceph-mon[29218]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:12.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:47:11] "GET /metrics HTTP/1.1" 200 37979 "" "Prometheus/2.33.4" 2024-03-21T12:47:12.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:11 smithi177 ceph-mon[25767]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:12.298 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:47:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:13 smithi172 ceph-mon[24813]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:13.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:13 smithi172 ceph-mon[29218]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:14.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:13 smithi177 ceph-mon[25767]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:47:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (5m) 2m ago 13m 24.9M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:47:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 2m ago 13m - - 2024-03-21T12:47:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (2m) 2m ago 12m 49.7M - 3.5 e1d6a67b021e f7cc7c5f0aef 2024-03-21T12:47:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283 running (8m) 2m ago 18m 513M - 19.0.0-2267-g960fbbe4 19e12293512f 41407098728c 2024-03-21T12:47:14.340 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (2m) 2m ago 19m 448M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (19m) 2m ago 19m 59.5M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (18m) 2m ago 18m 48.3M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (18m) 2m ago 18m 50.8M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (6m) 2m ago 14m 17.6M - 1.3.1 1dbe0e931976 1145f3da8c8f 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (5m) 2m ago 14m 17.5M - 1.3.1 1dbe0e931976 4437b302e23d 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (17m) 2m ago 17m 55.4M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (17m) 2m ago 17m 56.0M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (17m) 2m ago 17m 48.3M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (16m) 2m ago 16m 51.5M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (16m) 2m ago 16m 52.8M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (16m) 2m ago 16m 50.2M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (15m) 2m ago 15m 50.4M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (15m) 2m ago 15m 53.3M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (5m) 2m ago 14m 70.7M - 2.33.4 514e6a882f6e caf84b253e4b 2024-03-21T12:47:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (13m) 2m ago 13m 72.8M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:47:14.342 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (12m) 2m ago 12m 74.7M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:47:14.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:14 smithi172 ceph-mon[24813]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:14.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:14 smithi172 ceph-mon[29218]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:14.984 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T12:47:15.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:14 smithi177 ceph-mon[25767]: from='mgr.34772 172.21.15.177:0/4204224975' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:15 smithi172 ceph-mon[24813]: from='client.25114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:47:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:15 smithi172 ceph-mon[24813]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:15.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:15 smithi172 ceph-mon[29218]: from='client.25114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:47:15.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:15 smithi172 ceph-mon[29218]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:16.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:15 smithi177 ceph-mon[25767]: from='client.25114 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:47:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:15 smithi177 ceph-mon[25767]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:17.025 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:47:17.025 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:47:17.025 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:47:17.025 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:47:17.025 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:47:17.026 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:47:17.027 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:47:17.646 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T12:47:17.951 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:17 smithi172 ceph-mon[24813]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:17.952 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:17 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/818917711' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:47:17.952 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:17 smithi172 ceph-mon[29218]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:17.952 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:17 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/818917711' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:47:18.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:17 smithi177 ceph-mon[25767]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:18.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:17 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/818917711' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:47:19.690 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:47:19.690 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:47:19.690 INFO:teuthology.orchestra.run.smithi172.stdout: daemon grafana.a on smithi177 is in error state 2024-03-21T12:47:19.987 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:19 smithi172 ceph-mon[24813]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:19.987 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:19 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:19.987 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:19 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3316975970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:47:19.988 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:19 smithi172 ceph-mon[29218]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:19.988 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:19 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:19.988 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:19 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3316975970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:47:20.287 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph -s' 2024-03-21T12:47:20.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:19 smithi177 ceph-mon[25767]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:19 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:19 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3316975970' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:47:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:21 smithi177 ceph-mon[25767]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:22.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:47:21] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-21T12:47:22.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:21 smithi172 ceph-mon[24813]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:21 smithi172 ceph-mon[29218]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:22.361 INFO:teuthology.orchestra.run.smithi172.stdout: cluster: 2024-03-21T12:47:22.361 INFO:teuthology.orchestra.run.smithi172.stdout: id: 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:47:22.361 INFO:teuthology.orchestra.run.smithi172.stdout: health: HEALTH_WARN 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: 1 failed cephadm daemon(s) 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: services: 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: mgr: x(active, since 6m), standbys: y 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:47:22.362 INFO:teuthology.orchestra.run.smithi172.stdout: data: 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: pools: 6 pools, 161 pgs 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: objects: 209 objects, 585 KiB 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: usage: 90 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: pgs: 161 active+clean 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: io: 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T12:47:22.363 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:47:23.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:22 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1052738621' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:47:23.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:22 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1052738621' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:47:23.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph mgr fail' 2024-03-21T12:47:23.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:22 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1052738621' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:47:23.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:23 smithi172 ceph-mon[24813]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:23.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:23 smithi172 ceph-mon[29218]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:23 smithi177 ceph-mon[25767]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:47:26.136 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-mon[24813]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:26.136 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2300049077' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:47:26.136 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-mon[24813]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:47:25] ENGINE Bus STOPPING 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:47:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:47:25] ENGINE Bus STOPPED 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:47:25] ENGINE Bus STARTING 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:47:26] ENGINE Serving on http://:::9283 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:47:26] ENGINE Bus STARTED 2024-03-21T12:47:26.137 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-mon[29218]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:26.138 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2300049077' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:47:26.138 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:25 smithi172 ceph-mon[29218]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-mon[25767]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 90 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2300049077' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-mon[25767]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:25.879+0000 7fcc28349700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ignoring --setuser ceph since I am not root 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ignoring --setgroup ceph since I am not root 2024-03-21T12:47:26.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:25 smithi177 ceph-mgr[45987]: -- 172.21.15.177:0/3592720271 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55679c5ca5a0 con 0x55679c59ec00 2024-03-21T12:47:26.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 180' 2024-03-21T12:47:26.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:26.465+0000 7f7b8b870200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:47:26.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:26.562+0000 7f7b8b870200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:47:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2300049077' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:47:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: mgrmap e28: y(active, starting, since 0.804341s) 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:47:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2300049077' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: mgrmap e28: y(active, starting, since 0.804341s) 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:47:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:47:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: Manager daemon y is now available 2024-03-21T12:47:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:47:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:47:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:47:27.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2300049077' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:47:27.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: mgrmap e28: y(active, starting, since 0.804341s) 2024-03-21T12:47:27.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:47:27.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:47:27.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: Manager daemon y is now available 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:47:27.251 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:47:27.499 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:27.259+0000 7f7b8b870200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:47:27.500 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:27.354+0000 7f7b8b870200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:47:27.500 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:27.499+0000 7f7b8b870200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:47:27.826 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:27.603+0000 7f7b8b870200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:47:27.826 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:27.691+0000 7f7b8b870200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:47:28.089 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:27.970+0000 7f7b8b870200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:47:28.103 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-mon[25767]: mgrmap e29: y(active, since 1.82255s) 2024-03-21T12:47:28.103 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:47:26] ENGINE Bus STARTING 2024-03-21T12:47:28.104 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:47:27] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:47:28.104 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:47:27] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:47:28.104 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:27 smithi177 ceph-mon[25767]: [21/Mar/2024:12:47:27] ENGINE Bus STARTED 2024-03-21T12:47:28.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:28.088+0000 7f7b8b870200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:47:28.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:28.177+0000 7f7b8b870200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:47:28.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:28.275+0000 7f7b8b870200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[24813]: mgrmap e29: y(active, since 1.82255s) 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:47:26] ENGINE Bus STARTING 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:47:27] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:47:27] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[24813]: [21/Mar/2024:12:47:27] ENGINE Bus STARTED 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[29218]: mgrmap e29: y(active, since 1.82255s) 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:47:26] ENGINE Bus STARTING 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:47:27] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:47:27] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:47:28.369 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:27 smithi172 ceph-mon[29218]: [21/Mar/2024:12:47:27] ENGINE Bus STARTED 2024-03-21T12:47:28.785 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:28.478+0000 7f7b8b870200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:47:29.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:28 smithi172 ceph-mon[24813]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:29.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:28 smithi172 ceph-mon[24813]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:47:29.099 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:28 smithi172 ceph-mon[24813]: Cluster is now healthy 2024-03-21T12:47:29.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:28 smithi172 ceph-mon[29218]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:29.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:28 smithi172 ceph-mon[29218]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:47:29.099 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:28 smithi172 ceph-mon[29218]: Cluster is now healthy 2024-03-21T12:47:29.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-mon[25767]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:29.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-mon[25767]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:47:29.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:28 smithi177 ceph-mon[25767]: Cluster is now healthy 2024-03-21T12:47:29.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:29 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:29.490+0000 7f7b8b870200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:47:29.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:29 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:29.622+0000 7f7b8b870200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:47:30.029 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:29 smithi172 ceph-mon[24813]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:30.029 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:29 smithi172 ceph-mon[24813]: mgrmap e30: y(active, since 3s) 2024-03-21T12:47:30.029 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:29 smithi172 ceph-mon[29218]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:30.029 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:29 smithi172 ceph-mon[29218]: mgrmap e30: y(active, since 3s) 2024-03-21T12:47:30.284 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:29 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:29.973+0000 7f7b8b870200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:47:30.284 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:30.069+0000 7f7b8b870200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:47:30.284 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:30.283+0000 7f7b8b870200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:47:30.284 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:29 smithi177 ceph-mon[25767]: from='client.15198 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:47:30.284 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:29 smithi177 ceph-mon[25767]: mgrmap e30: y(active, since 3s) 2024-03-21T12:47:30.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:30.478+0000 7f7b8b870200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:47:31.061 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:30.783+0000 7f7b8b870200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:47:31.124 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:31 smithi172 ceph-mon[24813]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:31.124 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:31 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:31.124 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:31 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:31.315 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:31.061+0000 7f7b8b870200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:47:31.316 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:31 smithi177 ceph-mon[25767]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:31.316 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:31 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:31.316 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:31 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:31 smithi172 ceph-mon[29218]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:31 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:31 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:31.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:31.315+0000 7f7b8b870200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:47:32.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:32.130+0000 7f7b8b870200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:47:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:32.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:32 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.035 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:32.699+0000 7f7b8b870200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:47:33.035 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:32 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:32.799+0000 7f7b8b870200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:47:33.035 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:33.035+0000 7f7b8b870200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:47:33] ENGINE Bus STARTING 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CherryPy Checker: 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: The Application mounted at '' has an empty config. 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:47:33] ENGINE Serving on http://:::9283 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:47:33] ENGINE Bus STARTED 2024-03-21T12:47:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:47:33.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:33 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:47:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:47:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:47:33.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:33 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:47:34.473 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:47:34.473 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-21T12:47:34.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:47:34.475 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-21T12:47:34.476 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:34 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:47:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-21T12:47:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:34 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:47:35.496 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:47:35.496 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:47:35.496 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[24813]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:47:35.496 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[24813]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[29218]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:47:35.497 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:35 smithi172 ceph-mon[29218]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:47:35.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:35 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:47:35.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:35 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:47:35.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:35 smithi177 ceph-mon[25767]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:47:35.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:35 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:47:35.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:35 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:47:35.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:35 smithi177 ceph-mon[25767]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:47:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-mon[24813]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:47:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-mon[29218]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:47:37.107+0000 7f484871b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:37.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:37.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:37.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:37.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:37.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:37 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:37.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:37 smithi177 ceph-mon[25767]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.506 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:38.507 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:38.508 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:38.509 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:38.510 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-mon[25767]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:47:38.511 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-mon[25767]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:38 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:38.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:38.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:38.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:38.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:38 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:38.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:38.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:38.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:38.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:38.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:38.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:47:38.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:38.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:38 smithi172 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:38 smithi172 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:38 smithi172 ceph-mon[29218]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:47:38.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:38 smithi172 ceph-mon[29218]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:47:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:38.569Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.177:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.177:8765: connect: connection refused" 2024-03-21T12:47:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:38.569Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.177:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.177:8765: connect: connection refused" 2024-03-21T12:47:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:38.569Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.177:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.177:8765: connect: connection refused" 2024-03-21T12:47:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:38.571Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.177:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.177:8765: connect: connection refused" 2024-03-21T12:47:38.858 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:38.571Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.177:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.177:8765: connect: connection refused" 2024-03-21T12:47:39.301 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:39 smithi177 ceph-mon[25767]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T12:47:39.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:39 smithi172 ceph-mon[24813]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T12:47:39.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:39 smithi172 ceph-mon[29218]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-21T12:47:41.030 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:40 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:47:41.031 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:40 smithi177 bash[54971]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:47:41.322 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-mon[25767]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:41.323 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T12:47:41.323 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[49299]: ts=2024-03-21T12:47:41.031Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T12:47:41.324 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 podman[54984]: 2024-03-21 12:47:41.041595424 +0000 UTC m=+0.124760439 container died caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:47:41.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 podman[54984]: 2024-03-21 12:47:41.430362272 +0000 UTC m=+0.513527278 container cleanup caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:47:41.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 bash[54984]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T12:47:41.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 bash[55010]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:47:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:41 smithi172 ceph-mon[24813]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:41 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:41 smithi172 ceph-mon[29218]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:41 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 podman[54998]: 2024-03-21 12:47:41.673421891 +0000 UTC m=+0.634535065 container remove caf84b253e4b7555b749999f86f71e7f4e6d474b19d0dfacce257b97e3e48f42 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 systemd[1]: Stopped Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 2.029s CPU time. 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 podman[55084]: 2024-03-21 12:47:41.805691899 +0000 UTC m=+0.018242875 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T12:47:41.940 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:41 smithi177 podman[55084]: 2024-03-21 12:47:41.940941902 +0000 UTC m=+0.153492878 container create 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:47:42.204 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 podman[55084]: 2024-03-21 12:47:42.176804635 +0000 UTC m=+0.389355612 container init 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:47:42.204 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 podman[55084]: 2024-03-21 12:47:42.179643765 +0000 UTC m=+0.392194749 container start 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.205Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.205Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.205Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.205Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.205Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.205Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.206Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.207Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T12:47:42.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.207Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T12:47:42.609 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.210Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:47:42.609 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.210Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.834µs 2024-03-21T12:47:42.609 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.210Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:47:42.609 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 bash[55084]: 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e 2024-03-21T12:47:42.609 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:47:42.609 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.408Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.647Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.871Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.871Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.871Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.426µs wal_replay_duration=661.64855ms total_replay_duration=661.694448ms 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.877Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.877Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T12:47:42.899 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.877Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:47:42.900 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.899Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.804111ms db_storage=995ns remote_storage=1.602µs web_handler=484ns query_engine=1.113µs scrape=853.46µs scrape_sd=99.903µs notify=10.617µs notify_sd=17.338µs rules=20.214464ms 2024-03-21T12:47:42.900 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:47:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:47:42.899Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T12:47:43.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:43 smithi177 ceph-mon[25767]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:43.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:43 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:43.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:43 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:43.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:43 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:47:43.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:43 smithi177 ceph-mon[25767]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:47:43.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:43 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[24813]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[24813]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:47:43.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[29218]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:47:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[29218]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:47:43.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:43 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:47:45.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:45 smithi177 ceph-mon[25767]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:45.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:45 smithi172 ceph-mon[24813]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:45.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:45 smithi172 ceph-mon[29218]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:47:47.494 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:47:47.495 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:47 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:47:47.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:47 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:47:48.344 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[29218]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 385 B/s rd, 0 op/s 2024-03-21T12:47:48.344 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[29218]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:48.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:47:48.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:47:48.345 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[24813]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 385 B/s rd, 0 op/s 2024-03-21T12:47:48.345 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[24813]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:48.345 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:47:48.345 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:48 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:47:48.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:48 smithi177 ceph-mon[25767]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 385 B/s rd, 0 op/s 2024-03-21T12:47:48.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:48 smithi177 ceph-mon[25767]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:48.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:48 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:47:48.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:48 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:47:50.606 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-mon[24813]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-mon[29218]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:47:50.375+0000 7f484871b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:50.607 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:50.608 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:50.609 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:50.609 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:50.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:50 smithi177 ceph-mon[25767]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:50.610 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:50.611 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:47:50.612 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:50.613 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:51 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/4114976659' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:51 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:51.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:51.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:51.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:51.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:51 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/4114976659' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:47:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:51 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:51.648 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:51.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:51 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/4114976659' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:51 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:51.651 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:51.652 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:47:51.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi-foo-smithi172-bkzkmk 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: /usr/bin/podman: stderr Error: no such container ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:47:51.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:47:51.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:47:52.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:52 smithi177 ceph-mon[25767]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:52 smithi172 ceph-mon[24813]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:52.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:52 smithi172 ceph-mon[29218]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:47:54.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:54 smithi177 ceph-mon[25767]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2024-03-21T12:47:54.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:54 smithi172 ceph-mon[24813]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2024-03-21T12:47:54.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:54 smithi172 ceph-mon[29218]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2024-03-21T12:47:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:47:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:47:55] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-03-21T12:47:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:56 smithi177 ceph-mon[25767]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2024-03-21T12:47:56.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:56 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:56 smithi172 ceph-mon[24813]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2024-03-21T12:47:56.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:56 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:56.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:56 smithi172 ceph-mon[29218]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 237 B/s rd, 0 op/s 2024-03-21T12:47:56.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:56 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:47:58.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:47:58 smithi177 ceph-mon[25767]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-21T12:47:58.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:47:58 smithi172 ceph-mon[24813]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-21T12:47:58.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:47:58 smithi172 ceph-mon[29218]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-21T12:48:00.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:00 smithi177 ceph-mon[25767]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:48:00.645 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:00 smithi172 ceph-mon[24813]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:48:00.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:00 smithi172 ceph-mon[29218]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:48:01.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:01.645 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:01.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:02.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:02 smithi177 ceph-mon[25767]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:48:02.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:02 smithi172 ceph-mon[24813]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:48:02.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:02 smithi172 ceph-mon[29218]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-21T12:48:04.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:04 smithi177 ceph-mon[25767]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T12:48:04.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:04 smithi172 ceph-mon[24813]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T12:48:04.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:04 smithi172 ceph-mon[29218]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-21T12:48:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:48:05] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-21T12:48:06.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:06 smithi177 ceph-mon[25767]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:06.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:06 smithi172 ceph-mon[24813]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:06.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:06 smithi172 ceph-mon[29218]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:08 smithi177 ceph-mon[25767]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:08 smithi172 ceph-mon[24813]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:08 smithi172 ceph-mon[29218]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:10.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:10 smithi177 ceph-mon[25767]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:10 smithi172 ceph-mon[24813]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:10 smithi172 ceph-mon[29218]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:11.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:11.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:11 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:11 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:11 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:12.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:12 smithi177 ceph-mon[25767]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:12.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:12 smithi172 ceph-mon[24813]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:12.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:12 smithi172 ceph-mon[29218]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:14.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:14 smithi177 ceph-mon[25767]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:14.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:14 smithi172 ceph-mon[24813]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:14.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:14 smithi172 ceph-mon[29218]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:48:15] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-21T12:48:16.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:16 smithi177 ceph-mon[25767]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:16.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:16 smithi172 ceph-mon[24813]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:16 smithi172 ceph-mon[29218]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:18.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:18 smithi177 ceph-mon[25767]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:18 smithi172 ceph-mon[24813]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:18 smithi172 ceph-mon[29218]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:20 smithi177 ceph-mon[25767]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:20.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:20 smithi172 ceph-mon[24813]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:20.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:20 smithi172 ceph-mon[29218]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:21.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:21.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:22 smithi172 ceph-mon[24813]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:22.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:22 smithi172 ceph-mon[29218]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:22 smithi177 ceph-mon[25767]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:24.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:24 smithi172 ceph-mon[24813]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:24.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:24 smithi172 ceph-mon[29218]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:24.660 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:24 smithi177 ceph-mon[25767]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:48:25] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-21T12:48:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:26 smithi172 ceph-mon[24813]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:26 smithi172 ceph-mon[29218]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:26.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:26 smithi177 ceph-mon[25767]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:26.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:28 smithi172 ceph-mon[24813]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:28.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:28 smithi172 ceph-mon[29218]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:28.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:28 smithi177 ceph-mon[25767]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:30.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:30 smithi172 ceph-mon[24813]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:30.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:30 smithi172 ceph-mon[29218]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:30.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:30 smithi177 ceph-mon[25767]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:31.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:31 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:32.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:32 smithi172 ceph-mon[24813]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:32.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:32 smithi172 ceph-mon[29218]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:32.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:32 smithi177 ceph-mon[25767]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:34 smithi172 ceph-mon[24813]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:34.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:34 smithi172 ceph-mon[29218]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:34.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:34 smithi177 ceph-mon[25767]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:48:35] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-21T12:48:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:36 smithi172 ceph-mon[24813]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:36 smithi172 ceph-mon[29218]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:36 smithi177 ceph-mon[25767]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:38 smithi172 ceph-mon[24813]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:38.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:38 smithi172 ceph-mon[29218]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:38 smithi177 ceph-mon[25767]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:40.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:40 smithi172 ceph-mon[24813]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:40.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:40 smithi172 ceph-mon[29218]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:40 smithi177 ceph-mon[25767]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:41 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:41 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:41 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:42.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:42 smithi177 ceph-mon[25767]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:42.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:42 smithi172 ceph-mon[24813]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:42.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:42 smithi172 ceph-mon[29218]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:43.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:43 smithi177 ceph-mon[25767]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:43.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:43 smithi172 ceph-mon[24813]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:43.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:43 smithi172 ceph-mon[29218]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:45.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:45 smithi172 ceph-mon[24813]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:48:45] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-21T12:48:45.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:45 smithi172 ceph-mon[29218]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:45.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:45 smithi177 ceph-mon[25767]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:47 smithi177 ceph-mon[25767]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:47 smithi172 ceph-mon[24813]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:47 smithi172 ceph-mon[29218]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:50.025 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:49 smithi172 ceph-mon[24813]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:50.025 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:49 smithi172 ceph-mon[29218]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:50.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:49 smithi177 ceph-mon[25767]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:48:51.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:50 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:51.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:50 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:48:51.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:50 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:51.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:50 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:48:51.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:50 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:48:51.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:50 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:48:52.205 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:48:52.206 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:48:52.207 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.207 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:48:52.207 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:51 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:48:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:48:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:48:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:48:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:48:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:48:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:51 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:48:53.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[24813]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T12:48:53.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[24813]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T12:48:53.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:48:53.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:48:53.208 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[29218]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T12:48:53.209 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[29218]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T12:48:53.209 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:48:53.209 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:52 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:48:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:52 smithi177 ceph-mon[25767]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T12:48:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:52 smithi177 ceph-mon[25767]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T12:48:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:52 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:48:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:52 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:48:53.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:48:53.381+0000 7f484871b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:53.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:53.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:48:53.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:48:53.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:53.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:53.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:53 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.259 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.260 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:48:54.261 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:54.262 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.263 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:48:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:53 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:48:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:48:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:48:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:48:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:48:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:54 smithi172 ceph-mon[24813]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:55.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:54 smithi172 ceph-mon[29218]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:55.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:54 smithi177 ceph-mon[25767]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:48:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:48:55] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-21T12:48:57.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:56 smithi177 ceph-mon[25767]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:56 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:56 smithi172 ceph-mon[29218]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:56 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:56 smithi172 ceph-mon[24813]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:56 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:48:59.198 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:48:58 smithi172 ceph-mon[24813]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:59.199 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:48:58 smithi172 ceph-mon[29218]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:48:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:48:58 smithi177 ceph-mon[25767]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:49:01.355 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:00 smithi172 ceph-mon[24813]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:49:01.355 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:00 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:01.355 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:00 smithi172 ceph-mon[29218]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:49:01.355 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:00 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:01.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:00 smithi177 ceph-mon[25767]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:49:01.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:00 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:02 smithi177 ceph-mon[25767]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:49:03.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:02 smithi172 ceph-mon[24813]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:49:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:02 smithi172 ceph-mon[29218]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:49:05.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:04 smithi172 ceph-mon[24813]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:04 smithi172 ceph-mon[29218]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:05.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:04 smithi177 ceph-mon[25767]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:49:05] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-21T12:49:07.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:06 smithi177 ceph-mon[25767]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:06 smithi172 ceph-mon[24813]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:06 smithi172 ceph-mon[29218]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:09.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:08 smithi177 ceph-mon[25767]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:09.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:08 smithi172 ceph-mon[24813]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:09.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:08 smithi172 ceph-mon[29218]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:10 smithi172 ceph-mon[29218]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:10 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:10 smithi172 ceph-mon[24813]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:10 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:11.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:10 smithi177 ceph-mon[25767]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:10 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:11 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:12.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:11 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:11 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:13.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:12 smithi177 ceph-mon[25767]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:12 smithi172 ceph-mon[24813]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:13.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:12 smithi172 ceph-mon[29218]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:15.311 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:14 smithi172 ceph-mon[24813]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:15.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:14 smithi172 ceph-mon[29218]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:15.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:14 smithi177 ceph-mon[25767]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:49:15] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-21T12:49:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:16 smithi177 ceph-mon[25767]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:16 smithi172 ceph-mon[24813]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:16 smithi172 ceph-mon[29218]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:18 smithi177 ceph-mon[25767]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:18 smithi172 ceph-mon[24813]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:18 smithi172 ceph-mon[29218]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:21.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:20 smithi177 ceph-mon[25767]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:20 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:20 smithi172 ceph-mon[24813]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:20 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:20 smithi172 ceph-mon[29218]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:20 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:22 smithi177 ceph-mon[25767]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:23.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:22 smithi172 ceph-mon[24813]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:22 smithi172 ceph-mon[29218]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:25.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:24 smithi172 ceph-mon[24813]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:25.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:24 smithi172 ceph-mon[29218]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:24 smithi177 ceph-mon[25767]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:49:25] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-21T12:49:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:26 smithi177 ceph-mon[25767]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:26 smithi172 ceph-mon[24813]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:26 smithi172 ceph-mon[29218]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:28 smithi177 ceph-mon[25767]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:28 smithi172 ceph-mon[24813]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:28 smithi172 ceph-mon[29218]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:30 smithi177 ceph-mon[25767]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:30 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:30 smithi172 ceph-mon[24813]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:30 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:30 smithi172 ceph-mon[29218]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:30 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:33.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:32 smithi177 ceph-mon[25767]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:32 smithi172 ceph-mon[24813]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:32 smithi172 ceph-mon[29218]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:35.311 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:34 smithi172 ceph-mon[24813]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:35.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:34 smithi172 ceph-mon[29218]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:34 smithi177 ceph-mon[25767]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:49:35] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-21T12:49:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:36 smithi177 ceph-mon[25767]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:36 smithi172 ceph-mon[24813]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:36 smithi172 ceph-mon[29218]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:39.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:39 smithi177 ceph-mon[25767]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:39 smithi172 ceph-mon[24813]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:39 smithi172 ceph-mon[29218]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:41 smithi177 ceph-mon[25767]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:41 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:41 smithi172 ceph-mon[24813]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:41 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:41 smithi172 ceph-mon[29218]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:41 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:43.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:43 smithi177 ceph-mon[25767]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:43 smithi172 ceph-mon[24813]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:43 smithi172 ceph-mon[29218]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:45 smithi172 ceph-mon[24813]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:45.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:45 smithi172 ceph-mon[29218]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:45 smithi177 ceph-mon[25767]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:49:45] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-21T12:49:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:47 smithi177 ceph-mon[25767]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:47 smithi172 ceph-mon[24813]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:47 smithi172 ceph-mon[29218]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:49.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:49 smithi177 ceph-mon[25767]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:49 smithi172 ceph-mon[24813]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:49 smithi172 ceph-mon[29218]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:51.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:51 smithi177 ceph-mon[25767]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:51 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:51.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:51 smithi172 ceph-mon[24813]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:51.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:51 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:51 smithi172 ceph-mon[29218]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:49:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:53 smithi177 ceph-mon[25767]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:53 smithi172 ceph-mon[24813]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:53 smithi172 ceph-mon[29218]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:49:54.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:54 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:49:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:54 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:49:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:54 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:49:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:54 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:49:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:49:54.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:54 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:49:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:49:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 979 B/s rd, 0 op/s 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:49:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:55 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:49:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:49:55] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-21T12:49:56.040 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:49:55.930+0000 7f484871b700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:56.040 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:49:56.040 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:56.040 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:56.040 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:56.040 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:56.041 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:49:56.042 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:56.043 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:56.044 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:49:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:56 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:56 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:56 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:57 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:49:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:57 smithi177 ceph-mon[25767]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-03-21T12:49:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:57 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:49:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:49:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:57.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:57 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:57.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:57.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:49:57.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:49:57.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:57 smithi172 ceph-mon[29218]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-03-21T12:49:57.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:49:57.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:49:57.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:49:57.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:49:57.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:49:57.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:57 smithi172 ceph-mon[24813]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-03-21T12:49:59.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:49:59 smithi177 ceph-mon[25767]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:49:59 smithi172 ceph-mon[24813]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:49:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:49:59 smithi172 ceph-mon[29218]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:00 smithi177 ceph-mon[25767]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:50:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:00 smithi177 ceph-mon[25767]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:50:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:00 smithi177 ceph-mon[25767]: daemon grafana.a on smithi177 is in error state 2024-03-21T12:50:00.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:00 smithi172 ceph-mon[24813]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:50:00.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:00 smithi172 ceph-mon[24813]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:50:00.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:00 smithi172 ceph-mon[24813]: daemon grafana.a on smithi177 is in error state 2024-03-21T12:50:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:00 smithi172 ceph-mon[29218]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:50:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:00 smithi172 ceph-mon[29218]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:50:00.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:00 smithi172 ceph-mon[29218]: daemon grafana.a on smithi177 is in error state 2024-03-21T12:50:01.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:01 smithi177 ceph-mon[25767]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:01.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:01 smithi172 ceph-mon[24813]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:01.367 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:01.368 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:01 smithi172 ceph-mon[29218]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:01.368 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:03.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:02 smithi177 ceph-mon[25767]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:02 smithi172 ceph-mon[24813]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:03.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:02 smithi172 ceph-mon[29218]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:05 smithi177 ceph-mon[25767]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:50:05.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:05 smithi172 ceph-mon[24813]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:50:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:05 smithi172 ceph-mon[29218]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:50:05.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:50:05] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-03-21T12:50:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:07 smithi177 ceph-mon[25767]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:07 smithi172 ceph-mon[24813]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:07 smithi172 ceph-mon[29218]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:09.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:09 smithi177 ceph-mon[25767]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:09.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:09 smithi172 ceph-mon[24813]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:09.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:09 smithi172 ceph-mon[29218]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:11 smithi172 ceph-mon[24813]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:11 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:11 smithi172 ceph-mon[29218]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:11 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:11.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:11 smithi177 ceph-mon[25767]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:11 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:13.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:12 smithi177 ceph-mon[25767]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:13.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:12 smithi172 ceph-mon[24813]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:13.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:12 smithi172 ceph-mon[29218]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:15.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:15 smithi177 ceph-mon[25767]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:15 smithi172 ceph-mon[24813]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:50:15] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-21T12:50:15.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:15 smithi172 ceph-mon[29218]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:17 smithi177 ceph-mon[25767]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:17 smithi172 ceph-mon[24813]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:17 smithi172 ceph-mon[29218]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:19.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:19 smithi177 ceph-mon[25767]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:19 smithi172 ceph-mon[24813]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:19 smithi172 ceph-mon[29218]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:21 smithi177 ceph-mon[25767]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:21 smithi172 ceph-mon[24813]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:21 smithi172 ceph-mon[29218]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:23.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:22 smithi177 ceph-mon[25767]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:23.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:22 smithi172 ceph-mon[24813]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:23.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:22 smithi172 ceph-mon[29218]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:25.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:25 smithi177 ceph-mon[25767]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:25 smithi172 ceph-mon[24813]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:25.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:25 smithi172 ceph-mon[29218]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:25.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:50:25] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-21T12:50:26.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:26 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:26 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:26 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:27 smithi177 ceph-mon[25767]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:27 smithi172 ceph-mon[24813]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:27 smithi172 ceph-mon[29218]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:29 smithi177 ceph-mon[25767]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:29.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:29 smithi172 ceph-mon[24813]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:29.384 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:29 smithi172 ceph-mon[29218]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:29.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:50:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:31 smithi177 ceph-mon[25767]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:31 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:31.374 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:31 smithi172 ceph-mon[24813]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:31.374 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:31.374 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:31 smithi172 ceph-mon[29218]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:31.374 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:31.412 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:50:31.412 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (9m) 2m ago 17m 24.9M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:50:31.412 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 2m ago 16m - - 2024-03-21T12:50:31.412 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (6m) 2m ago 16m 52.7M - 3.5 e1d6a67b021e f7cc7c5f0aef 2024-03-21T12:50:31.412 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283 running (11m) 2m ago 21m 447M - 19.0.0-2267-g960fbbe4 19e12293512f 41407098728c 2024-03-21T12:50:31.412 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (6m) 2m ago 23m 474M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (23m) 2m ago 23m 60.6M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (21m) 2m ago 21m 46.3M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (21m) 2m ago 21m 46.9M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (9m) 2m ago 17m 17.2M - 1.3.1 1dbe0e931976 1145f3da8c8f 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (8m) 2m ago 17m 17.7M - 1.3.1 1dbe0e931976 4437b302e23d 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (21m) 2m ago 21m 55.7M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (20m) 2m ago 20m 56.3M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (20m) 2m ago 20m 48.6M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (20m) 2m ago 20m 51.8M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (19m) 2m ago 19m 52.2M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (19m) 2m ago 19m 50.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:50:31.413 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (18m) 2m ago 18m 50.9M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:50:31.414 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (18m) 2m ago 18m 54.2M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:50:31.414 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (2m) 2m ago 17m 44.1M - 2.33.4 514e6a882f6e 07be419a0ac5 2024-03-21T12:50:31.414 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (16m) 2m ago 16m 73.6M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:50:31.414 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (16m) 2m ago 16m 76.3M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:50:32.063 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T12:50:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:32 smithi177 ceph-mon[25767]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:50:32.390 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:32 smithi172 ceph-mon[24813]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:50:32.391 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:32 smithi172 ceph-mon[29218]: from='client.15297 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:50:33.356 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:33 smithi172 ceph-mon[24813]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:33.356 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:33 smithi172 ceph-mon[29218]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:33.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:33 smithi177 ceph-mon[25767]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:33.979 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:50:33.979 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:50:33.979 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:50:33.979 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:50:33.979 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:50:33.980 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:50:34.393 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:34 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2722669350' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:50:34.394 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:34 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2722669350' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:50:34.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph -s' 2024-03-21T12:50:34.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:34 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2722669350' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:50:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:35 smithi172 ceph-mon[24813]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:35.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:35 smithi172 ceph-mon[29218]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:35.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:35 smithi177 ceph-mon[25767]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:35.552 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:50:35] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.33.4" 2024-03-21T12:50:36.646 INFO:teuthology.orchestra.run.smithi172.stdout: cluster: 2024-03-21T12:50:36.646 INFO:teuthology.orchestra.run.smithi172.stdout: id: 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:50:36.646 INFO:teuthology.orchestra.run.smithi172.stdout: health: HEALTH_WARN 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: 1 failed cephadm daemon(s) 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: services: 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: mgr: y(active, since 3m), standbys: x 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: data: 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: pools: 6 pools, 161 pgs 2024-03-21T12:50:36.647 INFO:teuthology.orchestra.run.smithi172.stdout: objects: 209 objects, 585 KiB 2024-03-21T12:50:36.648 INFO:teuthology.orchestra.run.smithi172.stdout: usage: 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:36.648 INFO:teuthology.orchestra.run.smithi172.stdout: pgs: 161 active+clean 2024-03-21T12:50:36.648 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:50:36.648 INFO:teuthology.orchestra.run.smithi172.stdout: io: 2024-03-21T12:50:36.648 INFO:teuthology.orchestra.run.smithi172.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-21T12:50:36.648 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-21T12:50:37.252 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:37 smithi172 ceph-mon[24813]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:37.253 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:37 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3088775504' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:50:37.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:37 smithi172 ceph-mon[29218]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:37.253 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:37 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3088775504' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:50:37.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T12:50:37.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:37 smithi177 ceph-mon[25767]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:37.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:37 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3088775504' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T12:50:39.144 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:50:39.144 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:50:39.145 INFO:teuthology.orchestra.run.smithi172.stdout: daemon grafana.a on smithi177 is in error state 2024-03-21T12:50:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:39 smithi172 ceph-mon[24813]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:39 smithi172 ceph-mon[29218]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:39.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:39 smithi177 ceph-mon[25767]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:39.768 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-21T12:50:40.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:40 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/981293340' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:50:40.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:40 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/981293340' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:50:40.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:40 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/981293340' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:50:41.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:41 smithi172 ceph-mon[24813]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:41.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:41.344 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:41 smithi172 ceph-mon[24813]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:41.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:41 smithi172 ceph-mon[29218]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:41.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:41.345 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:41 smithi172 ceph-mon[29218]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:41 smithi177 ceph-mon[25767]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:41.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:41 smithi177 ceph-mon[25767]: from='mgr.15183 172.21.15.172:0/397777477' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:41.859 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:50:42.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph mgr fail' 2024-03-21T12:50:42.849 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:42 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3881480259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:50:42.849 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:42 smithi172 ceph-mon[24813]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:42.850 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:42 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3881480259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:50:42.850 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:42 smithi172 ceph-mon[29218]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:43.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:42 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3881480259' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:50:43.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:42 smithi177 ceph-mon[25767]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-mon[25767]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1490108833' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-mon[25767]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:50:45] ENGINE Bus STOPPING 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:50:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:50:45] ENGINE Bus STOPPED 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:50:45] ENGINE Bus STARTING 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:50:45] ENGINE Serving on http://:::9283 2024-03-21T12:50:45.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:50:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:50:45] ENGINE Bus STARTED 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mon[24813]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1490108833' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mon[24813]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:45.078+0000 7f48b606b700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ignoring --setuser ceph since I am not root 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ignoring --setgroup ceph since I am not root 2024-03-21T12:50:45.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mgr[63157]: -- 172.21.15.172:0/3222104486 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562a2bf6e5a0 con 0x562a2bf42c00 2024-03-21T12:50:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mon[29218]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:50:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1490108833' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-21T12:50:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-mon[29218]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T12:50:45.770 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:45.672+0000 7fa03068b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:50:45.770 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:45.770+0000 7fa03068b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:50:45.801 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 180' 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1490108833' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: mgrmap e32: x(active, starting, since 0.584096s) 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:50:46.086 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:50:46.087 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:50:46.087 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:50:46.087 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:50:46.087 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:50:46.087 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:50:46.087 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1490108833' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: mgrmap e32: x(active, starting, since 0.584096s) 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:50:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: Manager daemon x is now available 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:50:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:50:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:46 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: Manager daemon x is now available 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:50:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1490108833' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: mgrmap e32: x(active, starting, since 0.584096s) 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:50:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: Manager daemon x is now available 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:50:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-21T12:50:46.714 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:46.470+0000 7fa03068b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:50:46.715 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:46.564+0000 7fa03068b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:50:46.715 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:46.713+0000 7fa03068b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:50:47.030 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:46.811+0000 7fa03068b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:50:47.030 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:46.895+0000 7fa03068b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:50:47.196 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:47 smithi177 ceph-mon[25767]: mgrmap e33: x(active, since 1.5968s) 2024-03-21T12:50:47.196 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:47 smithi177 ceph-mon[25767]: [21/Mar/2024:12:50:46] ENGINE Bus STARTING 2024-03-21T12:50:47.197 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:47 smithi177 ceph-mon[25767]: [21/Mar/2024:12:50:46] ENGINE Serving on http://172.21.15.177:8765 2024-03-21T12:50:47.197 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:47 smithi177 ceph-mon[25767]: [21/Mar/2024:12:50:46] ENGINE Serving on https://172.21.15.177:7150 2024-03-21T12:50:47.197 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:47 smithi177 ceph-mon[25767]: [21/Mar/2024:12:50:46] ENGINE Bus STARTED 2024-03-21T12:50:47.288 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:47.169+0000 7fa03068b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[24813]: mgrmap e33: x(active, since 1.5968s) 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[24813]: [21/Mar/2024:12:50:46] ENGINE Bus STARTING 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[24813]: [21/Mar/2024:12:50:46] ENGINE Serving on http://172.21.15.177:8765 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[24813]: [21/Mar/2024:12:50:46] ENGINE Serving on https://172.21.15.177:7150 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[24813]: [21/Mar/2024:12:50:46] ENGINE Bus STARTED 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[29218]: mgrmap e33: x(active, since 1.5968s) 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[29218]: [21/Mar/2024:12:50:46] ENGINE Bus STARTING 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[29218]: [21/Mar/2024:12:50:46] ENGINE Serving on http://172.21.15.177:8765 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[29218]: [21/Mar/2024:12:50:46] ENGINE Serving on https://172.21.15.177:7150 2024-03-21T12:50:47.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-mon[29218]: [21/Mar/2024:12:50:46] ENGINE Bus STARTED 2024-03-21T12:50:47.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:47.287+0000 7fa03068b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:50:47.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:47.380+0000 7fa03068b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:50:47.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:47.477+0000 7fa03068b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:50:47.940 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:47.657+0000 7fa03068b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:50:48.347 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[24813]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:48.347 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[24813]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:50:48.347 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[24813]: Cluster is now healthy 2024-03-21T12:50:48.347 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[24813]: mgrmap e34: x(active, since 2s) 2024-03-21T12:50:48.347 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[29218]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:48.347 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[29218]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:50:48.348 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[29218]: Cluster is now healthy 2024-03-21T12:50:48.348 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-mon[29218]: mgrmap e34: x(active, since 2s) 2024-03-21T12:50:48.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:48 smithi177 ceph-mon[25767]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:48.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:48 smithi177 ceph-mon[25767]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:50:48.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:48 smithi177 ceph-mon[25767]: Cluster is now healthy 2024-03-21T12:50:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:48 smithi177 ceph-mon[25767]: mgrmap e34: x(active, since 2s) 2024-03-21T12:50:49.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:48.669+0000 7fa03068b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:50:49.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:48.803+0000 7fa03068b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:50:49.483 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:49.178+0000 7fa03068b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:50:49.483 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:49.274+0000 7fa03068b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:50:49.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:49.482+0000 7fa03068b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:50:49.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:49.675+0000 7fa03068b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:50:50.254 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:49.992+0000 7fa03068b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:50:50.254 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[24813]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:50.254 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[24813]: mgrmap e35: x(active, since 4s) 2024-03-21T12:50:50.254 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:50.254 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:50.254 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[29218]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:50.255 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[29218]: mgrmap e35: x(active, since 4s) 2024-03-21T12:50:50.255 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:50.255 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:50.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:50 smithi177 ceph-mon[25767]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:50 smithi177 ceph-mon[25767]: mgrmap e35: x(active, since 4s) 2024-03-21T12:50:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:50 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:50 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:50.512 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:50.265+0000 7fa03068b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:50:50.816 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:50 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:50.511+0000 7fa03068b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:50:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:51 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:51 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:51 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:51.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:51.387+0000 7fa03068b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:50:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:50:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:51.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:51 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.327 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:52.000+0000 7fa03068b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:50:52.327 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:52.109+0000 7fa03068b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:50:52.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:52 smithi177 ceph-mon[25767]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:52.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:52 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:52 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-mon[24813]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-mon[29218]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:52.345+0000 7fa03068b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:50:52.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:50:52] ENGINE Bus STARTING 2024-03-21T12:50:52.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CherryPy Checker: 2024-03-21T12:50:52.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: The Application mounted at '' has an empty config. 2024-03-21T12:50:52.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:50:52.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:50:52] ENGINE Serving on http://:::9283 2024-03-21T12:50:52.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:50:52] ENGINE Bus STARTED 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Standby manager daemon y started 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:50:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:50:53.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:50:53.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:50:53.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:50:53.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:53 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Standby manager daemon y started 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:53.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Standby manager daemon y started 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-21T12:50:53.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:50:53.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:53 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:50:54.571 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:50:54.571 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:50:54.571 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:50:54.571 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:50:54.572 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:54 smithi177 ceph-mon[25767]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:50:54.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:50:54.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:50:54.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:50:54.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:54 smithi172 ceph-mon[29218]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:50:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:50:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:50:55] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-21T12:50:56.674 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-mon[25767]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:50:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:56 smithi172 ceph-mon[24813]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:50:56.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:56 smithi172 ceph-mon[29218]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:50:56.978 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:50:56.979 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 bash[57935]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.978Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T12:50:57.236 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T12:50:57.237 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[55098]: ts=2024-03-21T12:50:56.979Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T12:50:57.237 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:56 smithi177 podman[57946]: 2024-03-21 12:50:56.989143462 +0000 UTC m=+0.122970318 container died 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:50:57.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 podman[57946]: 2024-03-21 12:50:57.403062189 +0000 UTC m=+0.536889041 container cleanup 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:50:57.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 bash[57946]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T12:50:57.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 bash[57980]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 podman[57960]: 2024-03-21 12:50:57.645873196 +0000 UTC m=+0.659160573 container remove 07be419a0ac50d9673adc61146d9b3dd05aeff88dd59af8eed892955a266c24e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 systemd[1]: Stopped Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.381s CPU time. 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 podman[58052]: 2024-03-21 12:50:57.780857068 +0000 UTC m=+0.020123021 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T12:50:57.930 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:57 smithi177 podman[58052]: 2024-03-21 12:50:57.930384308 +0000 UTC m=+0.169650262 container create d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:50:58.186 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 podman[58052]: 2024-03-21 12:50:58.140792926 +0000 UTC m=+0.380058890 container init d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:50:58.186 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 podman[58052]: 2024-03-21 12:50:58.143642574 +0000 UTC m=+0.382908533 container start d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.168Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.168Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.168Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.168Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.168Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.168Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.169Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.170Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.170Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.174Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.174Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.827µs 2024-03-21T12:50:58.187 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.174Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:50:58.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 bash[58052]: d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 2024-03-21T12:50:58.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:50:58.608 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.390Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-03-21T12:50:58.900 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-mon[25767]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:50:58.900 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:58.901 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:58.901 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:50:58.901 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:50:58.901 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.626Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-03-21T12:50:58.901 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.847Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-03-21T12:50:58.901 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.901Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-03-21T12:50:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[24813]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:50:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:50:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:50:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[29218]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:50:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:50:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:50:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:58 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:50:59.170 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.901Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-03-21T12:50:59.170 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.901Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=36.612µs wal_replay_duration=727.294854ms total_replay_duration=727.344735ms 2024-03-21T12:50:59.170 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.906Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:50:59.170 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.906Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T12:50:59.170 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.906Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:50:59.171 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.928Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.877944ms db_storage=818ns remote_storage=1.524µs web_handler=404ns query_engine=838ns scrape=373.144µs scrape_sd=127.134µs notify=23.195µs notify_sd=18.813µs rules=21.006561ms 2024-03-21T12:50:59.171 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:50:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:50:58.928Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T12:50:59.948 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:50:59 smithi177 ceph-mon[25767]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:51:00.047 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:50:59 smithi172 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:51:00.048 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:50:59 smithi172 ceph-mon[29218]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:51:01.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:00 smithi172 ceph-mon[24813]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:51:01.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:01.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:00 smithi172 ceph-mon[29218]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:51:01.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:01.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:00 smithi177 ceph-mon[25767]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:51:01.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:01.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:01.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:02.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:03.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:02 smithi177 ceph-mon[25767]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:51:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:02 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:02 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:02 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:51:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:02 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:51:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:02 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[24813]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:51:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:51:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:51:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[29218]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:51:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:51:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:51:03.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:02 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:51:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:03 smithi177 ceph-mon[25767]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:51:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:03 smithi177 ceph-mon[25767]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:03 smithi172 ceph-mon[24813]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:51:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:03 smithi172 ceph-mon[24813]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:03 smithi172 ceph-mon[29218]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:51:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:03 smithi172 ceph-mon[29218]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:05 smithi177 ceph-mon[25767]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:05 smithi172 ceph-mon[24813]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:05 smithi172 ceph-mon[29218]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:08.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:07 smithi177 ceph-mon[25767]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-21T12:51:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:07 smithi172 ceph-mon[24813]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-21T12:51:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:07 smithi172 ceph-mon[29218]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-21T12:51:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:09 smithi177 ceph-mon[25767]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:10.142 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:09 smithi172 ceph-mon[24813]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:10.142 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:09 smithi172 ceph-mon[29218]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:12.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:51:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:51:11] "GET /metrics HTTP/1.1" 200 37991 "" "Prometheus/2.33.4" 2024-03-21T12:51:12.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:11 smithi177 ceph-mon[25767]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-21T12:51:12.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:11 smithi172 ceph-mon[24813]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-21T12:51:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:11 smithi172 ceph-mon[29218]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 590 B/s rd, 0 op/s 2024-03-21T12:51:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:13 smithi177 ceph-mon[25767]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:51:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:13 smithi172 ceph-mon[24813]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:51:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:13 smithi172 ceph-mon[29218]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:51:16.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:15 smithi177 ceph-mon[25767]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:15 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:16.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:15 smithi172 ceph-mon[24813]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:16.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:15 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:15 smithi172 ceph-mon[29218]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:15 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:18.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:17 smithi177 ceph-mon[25767]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:17 smithi172 ceph-mon[24813]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:17 smithi172 ceph-mon[29218]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:20.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:19 smithi177 ceph-mon[25767]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:20.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:19 smithi172 ceph-mon[24813]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:19 smithi172 ceph-mon[29218]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:22.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:51:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:51:21] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-21T12:51:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:21 smithi177 ceph-mon[25767]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:21 smithi172 ceph-mon[24813]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:21 smithi172 ceph-mon[29218]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:24.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:23 smithi177 ceph-mon[25767]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:24.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:23 smithi172 ceph-mon[24813]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:24.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:23 smithi172 ceph-mon[29218]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:25 smithi177 ceph-mon[25767]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:25 smithi172 ceph-mon[24813]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:25 smithi172 ceph-mon[29218]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:27 smithi177 ceph-mon[25767]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:28.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:27 smithi172 ceph-mon[24813]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:27 smithi172 ceph-mon[29218]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:30.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:29 smithi177 ceph-mon[25767]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:29 smithi172 ceph-mon[24813]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:29 smithi172 ceph-mon[29218]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:31.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:30 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:31.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:30 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:31.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:30 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:32.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:51:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:51:31] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-03-21T12:51:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:31 smithi177 ceph-mon[25767]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:31 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:31 smithi172 ceph-mon[24813]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:31 smithi172 ceph-mon[29218]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:33 smithi177 ceph-mon[25767]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:33 smithi172 ceph-mon[24813]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:33 smithi172 ceph-mon[29218]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:35 smithi177 ceph-mon[25767]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:35 smithi172 ceph-mon[24813]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:35 smithi172 ceph-mon[29218]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:38.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:37 smithi177 ceph-mon[25767]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:38.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:37 smithi172 ceph-mon[24813]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:37 smithi172 ceph-mon[29218]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:39 smithi177 ceph-mon[25767]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:39 smithi172 ceph-mon[24813]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:39 smithi172 ceph-mon[29218]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:42.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:51:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:51:41] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-03-21T12:51:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:41 smithi177 ceph-mon[25767]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:42.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:41 smithi172 ceph-mon[24813]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:41 smithi172 ceph-mon[29218]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:43 smithi177 ceph-mon[25767]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:43 smithi172 ceph-mon[24813]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:43 smithi172 ceph-mon[29218]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:46.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:45 smithi177 ceph-mon[25767]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:45 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:45 smithi172 ceph-mon[29218]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:45 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:45 smithi172 ceph-mon[24813]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:45 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:51:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:47 smithi177 ceph-mon[25767]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:47 smithi172 ceph-mon[24813]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:47 smithi172 ceph-mon[29218]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:50.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:49 smithi177 ceph-mon[25767]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:49 smithi172 ceph-mon[24813]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:49 smithi172 ceph-mon[29218]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:51:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:51:51] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.33.4" 2024-03-21T12:51:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:51 smithi177 ceph-mon[25767]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:51 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:51 smithi172 ceph-mon[24813]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:51 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:51 smithi172 ceph-mon[29218]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:51:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:53 smithi177 ceph-mon[25767]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:53 smithi172 ceph-mon[24813]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:53 smithi172 ceph-mon[29218]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:51:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:55 smithi177 ceph-mon[25767]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:56.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:55 smithi172 ceph-mon[24813]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:55 smithi172 ceph-mon[29218]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:58.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:57 smithi177 ceph-mon[25767]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:58.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:57 smithi172 ceph-mon[24813]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:51:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:57 smithi172 ceph-mon[29218]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:51:59 smithi177 ceph-mon[25767]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:51:59 smithi172 ceph-mon[24813]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:51:59 smithi172 ceph-mon[29218]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:02.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:52:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:52:01] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-21T12:52:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:01 smithi177 ceph-mon[25767]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:01 smithi172 ceph-mon[24813]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:01 smithi172 ceph-mon[29218]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:02 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:52:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:02 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:52:03.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:02 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:52:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:03 smithi177 ceph-mon[25767]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:03 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:52:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:03 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:52:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:03 smithi177 ceph-mon[25767]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:52:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:03 smithi177 ceph-mon[25767]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:03 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:52:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[24813]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:52:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:52:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[24813]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:52:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[24813]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:52:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[29218]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:52:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:52:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[29218]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-21T12:52:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[29218]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:03 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:52:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:05 smithi172 ceph-mon[24813]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:05 smithi172 ceph-mon[29218]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:05 smithi177 ceph-mon[25767]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:08.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:07 smithi177 ceph-mon[25767]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:07 smithi172 ceph-mon[24813]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:07 smithi172 ceph-mon[29218]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:09 smithi172 ceph-mon[24813]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:09 smithi172 ceph-mon[29218]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:10.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:09 smithi177 ceph-mon[25767]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:52:12.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:52:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:52:11] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-21T12:52:12.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:12.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:11 smithi177 ceph-mon[25767]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:52:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:11 smithi172 ceph-mon[24813]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:52:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:11 smithi172 ceph-mon[29218]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 587 B/s rd, 0 op/s 2024-03-21T12:52:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:13 smithi172 ceph-mon[24813]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:52:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:13 smithi172 ceph-mon[29218]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:52:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:13 smithi177 ceph-mon[25767]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:52:16.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:15 smithi172 ceph-mon[24813]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:16.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:15 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:16.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:15 smithi172 ceph-mon[29218]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:16.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:15 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:15 smithi177 ceph-mon[25767]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:15 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:17 smithi177 ceph-mon[25767]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:17 smithi172 ceph-mon[24813]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:17 smithi172 ceph-mon[29218]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:20.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:19 smithi172 ceph-mon[24813]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:19 smithi172 ceph-mon[29218]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:20.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:19 smithi177 ceph-mon[25767]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:22.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:52:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:52:21] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-21T12:52:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:21 smithi177 ceph-mon[25767]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:21 smithi172 ceph-mon[24813]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:21 smithi172 ceph-mon[29218]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:24.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:23 smithi172 ceph-mon[24813]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:24.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:23 smithi172 ceph-mon[29218]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:24.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:23 smithi177 ceph-mon[25767]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:25 smithi172 ceph-mon[24813]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:25 smithi172 ceph-mon[29218]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:26.160 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:25 smithi177 ceph-mon[25767]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:27 smithi177 ceph-mon[25767]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:28.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:27 smithi172 ceph-mon[24813]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:27 smithi172 ceph-mon[29218]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:29 smithi172 ceph-mon[24813]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:29 smithi172 ceph-mon[29218]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:29 smithi177 ceph-mon[25767]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:31.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:30 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:31.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:30 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:31.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:30 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:31 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:31 smithi177 ceph-mon[25767]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:32.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:52:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:52:31] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-21T12:52:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:31 smithi172 ceph-mon[24813]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:32.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:31 smithi172 ceph-mon[29218]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:33 smithi172 ceph-mon[24813]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:33 smithi172 ceph-mon[29218]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:33 smithi177 ceph-mon[25767]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:35 smithi172 ceph-mon[24813]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:35 smithi172 ceph-mon[29218]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:36.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:35 smithi177 ceph-mon[25767]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:38.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:37 smithi177 ceph-mon[25767]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:38.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:37 smithi172 ceph-mon[24813]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:38.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:37 smithi172 ceph-mon[29218]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:39 smithi172 ceph-mon[24813]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:39 smithi172 ceph-mon[29218]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:39 smithi177 ceph-mon[25767]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:42.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:42.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:41 smithi177 ceph-mon[25767]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:42.110 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:52:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:52:41] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-21T12:52:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:41 smithi172 ceph-mon[24813]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:41 smithi172 ceph-mon[29218]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:43 smithi172 ceph-mon[24813]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:43 smithi172 ceph-mon[29218]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:43 smithi177 ceph-mon[25767]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:45 smithi172 ceph-mon[24813]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:45 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:45 smithi172 ceph-mon[29218]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:45 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:45 smithi177 ceph-mon[25767]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:45 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:52:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:47 smithi177 ceph-mon[25767]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:47 smithi172 ceph-mon[24813]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:47 smithi172 ceph-mon[29218]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:50.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:49 smithi172 ceph-mon[24813]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:50.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:49 smithi172 ceph-mon[29218]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:49 smithi177 ceph-mon[25767]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:51 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:51 smithi177 ceph-mon[25767]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:52:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:52:51] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-21T12:52:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:51 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:51 smithi172 ceph-mon[24813]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:52:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:51 smithi172 ceph-mon[29218]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:53 smithi172 ceph-mon[24813]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:53 smithi172 ceph-mon[29218]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:53 smithi177 ceph-mon[25767]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:56.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:55 smithi172 ceph-mon[24813]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:56.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:55 smithi172 ceph-mon[29218]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:56.159 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:55 smithi177 ceph-mon[25767]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:52:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:57 smithi177 ceph-mon[25767]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:58.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:57 smithi172 ceph-mon[24813]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:52:58.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:57 smithi172 ceph-mon[29218]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:52:59 smithi172 ceph-mon[24813]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:52:59 smithi172 ceph-mon[29218]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:00.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:52:59 smithi177 ceph-mon[25767]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:01 smithi177 ceph-mon[25767]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:02.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:53:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:53:01] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-21T12:53:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:01 smithi172 ceph-mon[24813]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:01 smithi172 ceph-mon[29218]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:04.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:03 smithi177 ceph-mon[25767]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:03 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:03 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:53:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:03 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:53:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:03 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:53:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[24813]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:53:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:53:04.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:53:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[29218]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:53:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:53:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:53:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:03 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:53:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:04 smithi177 ceph-mon[25767]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-21T12:53:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:04 smithi177 ceph-mon[25767]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:05.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:04 smithi172 ceph-mon[24813]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-21T12:53:05.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:04 smithi172 ceph-mon[24813]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:04 smithi172 ceph-mon[29218]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-21T12:53:05.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:04 smithi172 ceph-mon[29218]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:06 smithi177 ceph-mon[25767]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-21T12:53:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:06 smithi172 ceph-mon[24813]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-21T12:53:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:06 smithi172 ceph-mon[29218]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-21T12:53:09.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:08 smithi177 ceph-mon[25767]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:09.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:08 smithi172 ceph-mon[24813]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:09.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:08 smithi172 ceph-mon[29218]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:10 smithi177 ceph-mon[25767]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:10 smithi172 ceph-mon[24813]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:11.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:10 smithi172 ceph-mon[29218]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:12.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:12.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:53:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:53:11] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-21T12:53:12.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:12 smithi177 ceph-mon[25767]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:12 smithi172 ceph-mon[24813]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:13.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:12 smithi172 ceph-mon[29218]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:14 smithi177 ceph-mon[25767]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:53:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:14 smithi172 ceph-mon[24813]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:53:15.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:14 smithi172 ceph-mon[29218]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:53:16.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:15 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:15 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:15 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:16 smithi177 ceph-mon[25767]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:16 smithi172 ceph-mon[24813]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:16 smithi172 ceph-mon[29218]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:18 smithi177 ceph-mon[25767]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:18 smithi172 ceph-mon[24813]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:18 smithi172 ceph-mon[29218]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:20 smithi177 ceph-mon[25767]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:20 smithi172 ceph-mon[24813]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:20 smithi172 ceph-mon[29218]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:22.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:53:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:53:21] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-21T12:53:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:22 smithi177 ceph-mon[25767]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:23.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:22 smithi172 ceph-mon[24813]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:22 smithi172 ceph-mon[29218]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:24 smithi177 ceph-mon[25767]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:24 smithi172 ceph-mon[24813]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:25.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:24 smithi172 ceph-mon[29218]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:26 smithi177 ceph-mon[25767]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:26 smithi172 ceph-mon[24813]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:26 smithi172 ceph-mon[29218]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:28 smithi177 ceph-mon[25767]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:28 smithi172 ceph-mon[24813]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:28 smithi172 ceph-mon[29218]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:30 smithi177 ceph-mon[25767]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:30 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:30 smithi172 ceph-mon[24813]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:30 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:31.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:30 smithi172 ceph-mon[29218]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:31.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:30 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:31 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:32.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:53:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:53:31] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.33.4" 2024-03-21T12:53:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:32.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:32 smithi177 ceph-mon[25767]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:32 smithi172 ceph-mon[24813]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:32 smithi172 ceph-mon[29218]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:34 smithi177 ceph-mon[25767]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:35.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:34 smithi172 ceph-mon[24813]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:35.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:34 smithi172 ceph-mon[29218]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:36 smithi177 ceph-mon[25767]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:36 smithi172 ceph-mon[24813]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:36 smithi172 ceph-mon[29218]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:38 smithi177 ceph-mon[25767]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:38 smithi172 ceph-mon[24813]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:39.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:38 smithi172 ceph-mon[29218]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:40 smithi177 ceph-mon[25767]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:40 smithi172 ceph-mon[24813]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:40 smithi172 ceph-mon[29218]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:42.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:53:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:53:41] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.33.4" 2024-03-21T12:53:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:43.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:42 smithi177 ceph-mon[25767]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:42 smithi172 ceph-mon[24813]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:42 smithi172 ceph-mon[29218]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:44 smithi177 ceph-mon[25767]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:44 smithi172 ceph-mon[24813]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:45.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:44 smithi172 ceph-mon[29218]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:45 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:45 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:45 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:53:47.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:46 smithi177 ceph-mon[25767]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:46 smithi172 ceph-mon[24813]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:46 smithi172 ceph-mon[29218]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:48.782 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:53:49.094 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:48 smithi172 ceph-mon[24813]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:49.094 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:48 smithi172 ceph-mon[29218]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:49.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:48 smithi177 ceph-mon[25767]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:50.741 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (12m) 2m ago 20m 24.8M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 2m ago 20m - - 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 error 2m ago 19m 56.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283 running (15m) 2m ago 24m 495M - 19.0.0-2267-g960fbbe4 19e12293512f 41407098728c 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (9m) 2m ago 26m 74.0M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (26m) 2m ago 26m 68.8M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (25m) 2m ago 25m 52.3M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (25m) 2m ago 25m 51.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (12m) 2m ago 21m 19.5M - 1.3.1 1dbe0e931976 1145f3da8c8f 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (12m) 2m ago 20m 18.0M - 1.3.1 1dbe0e931976 4437b302e23d 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (24m) 2m ago 24m 55.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (24m) 2m ago 24m 56.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (23m) 2m ago 23m 48.7M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:53:50.742 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (23m) 2m ago 23m 52.0M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (23m) 2m ago 23m 52.4M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (22m) 2m ago 22m 50.9M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (22m) 2m ago 22m 51.3M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (21m) 2m ago 21m 54.5M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (2m) 2m ago 21m 44.2M - 2.33.4 514e6a882f6e d11f0aa24855 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (19m) 2m ago 19m 74.7M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:53:50.743 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (19m) 2m ago 19m 77.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:53:51.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:50 smithi172 ceph-mon[24813]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:51.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:50 smithi172 ceph-mon[29218]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:51.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ls' 2024-03-21T12:53:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:50 smithi177 ceph-mon[25767]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:52.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:53:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:53:51] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-21T12:53:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:52 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:52 smithi177 ceph-mon[25767]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:52 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:52 smithi172 ceph-mon[24813]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:52 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:53:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:52 smithi172 ceph-mon[29218]: from='client.15363 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:53.291 INFO:teuthology.orchestra.run.smithi172.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-21T12:53:53.291 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager ?:9093,9094 1/1 3m ago 21m smithi172=a;count:1 2024-03-21T12:53:53.291 INFO:teuthology.orchestra.run.smithi172.stdout:grafana ?:3000 0/1 2m ago 21m smithi177=a;count:1 2024-03-21T12:53:53.291 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo ?:5000 0/1 3m ago 19m count:1 2024-03-21T12:53:53.292 INFO:teuthology.orchestra.run.smithi172.stdout:mgr 2/2 3m ago 24m smithi172=y;smithi177=x;count:2 2024-03-21T12:53:53.292 INFO:teuthology.orchestra.run.smithi172.stdout:mon 3/3 3m ago 25m smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi177:172.21.15.177=b;count:3 2024-03-21T12:53:53.292 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter ?:9100 2/2 3m ago 21m smithi172=a;smithi177=b;count:2 2024-03-21T12:53:53.292 INFO:teuthology.orchestra.run.smithi172.stdout:osd 8 3m ago - 2024-03-21T12:53:53.292 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus ?:9095 1/1 2m ago 21m smithi177=a;count:1 2024-03-21T12:53:53.292 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo ?:8000 2/2 3m ago 19m count:2 2024-03-21T12:53:53.292 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:53 smithi172 ceph-mon[24813]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:53.292 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:53 smithi172 ceph-mon[29218]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:53 smithi177 ceph-mon[25767]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:53.906 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T12:53:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:54 smithi172 ceph-mon[29218]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:54.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:54 smithi172 ceph-mon[29218]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:54 smithi172 ceph-mon[24813]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:54 smithi172 ceph-mon[24813]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:54 smithi177 ceph-mon[25767]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:54 smithi177 ceph-mon[25767]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:53:55.904 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:53:55.905 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:53:56.630 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-21T12:53:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:56 smithi177 ceph-mon[25767]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:56 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2001366951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:53:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:56 smithi172 ceph-mon[24813]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:56 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2001366951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:53:56.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:56 smithi172 ceph-mon[29218]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:53:56.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:56 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2001366951' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:53:59.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:58 smithi177 ceph-mon[25767]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:58 smithi172 ceph-mon[24813]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:58 smithi172 ceph-mon[29218]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:53:59.875 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:53:59 smithi172 ceph-mon[24813]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:53:59.876 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:53:59 smithi172 ceph-mon[29218]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:53:59 smithi177 ceph-mon[25767]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:00.659 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:00.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:00 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:00 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:01.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T12:54:02.007 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:02.007 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[24813]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:02.007 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[24813]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-21T12:54:02.007 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[24813]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-21T12:54:02.007 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[24813]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:02.008 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:02.008 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[29218]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:02.008 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[29218]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-21T12:54:02.008 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[29218]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-21T12:54:02.008 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:01 smithi172 ceph-mon[29218]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:02.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:54:01] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.33.4" 2024-03-21T12:54:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:01 smithi177 ceph-mon[25767]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:01 smithi177 ceph-mon[25767]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-21T12:54:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:01 smithi177 ceph-mon[25767]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-21T12:54:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:01 smithi177 ceph-mon[25767]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:54:04.052 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:03 smithi172 ceph-mon[24813]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:54:04.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:03 smithi172 ceph-mon[29218]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:54:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:03 smithi177 ceph-mon[25767]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:54:04.723 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:05.330+0000 7f7b7f27a700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ignoring --setuser ceph since I am not root 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: ignoring --setgroup ceph since I am not root 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mgr[45987]: -- 172.21.15.177:0/590443020 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557fb6f105a0 con 0x557fb6ee4c00 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:54:05.588 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: Failing over to other MGR 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 714 B/s rd, 0 op/s 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:05.589 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:54:05.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:54:05] ENGINE Bus STOPPING 2024-03-21T12:54:05.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:54:05] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:54:05] ENGINE Bus STOPPED 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:54:05] ENGINE Bus STARTING 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:54:05] ENGINE Serving on http://:::9283 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:54:05] ENGINE Bus STARTED 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T12:54:05.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: Failing over to other MGR 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 714 B/s rd, 0 op/s 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-21T12:54:05.595 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: Failing over to other MGR 2024-03-21T12:54:05.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: from='mgr.34982 172.21.15.177:0/3401316415' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T12:54:05.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-21T12:54:05.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:54:05.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 714 B/s rd, 0 op/s 2024-03-21T12:54:05.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:05.596 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:05 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:54:06.335 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:05.909+0000 7f4aa537d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:54:06.335 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:06.007+0000 7f4aa537d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:54:06.367 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (12m) 3m ago 20m 24.8M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 3m ago 20m - - 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 error 3m ago 19m 56.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283 running (15m) 3m ago 25m 495M - 19.0.0-2267-g960fbbe4 19e12293512f 41407098728c 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (9m) 3m ago 26m 74.0M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (26m) 3m ago 26m 68.8M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (25m) 3m ago 25m 52.3M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (25m) 3m ago 25m 51.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (12m) 3m ago 21m 19.5M - 1.3.1 1dbe0e931976 1145f3da8c8f 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (12m) 3m ago 21m 18.0M - 1.3.1 1dbe0e931976 4437b302e23d 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (24m) 3m ago 24m 55.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (24m) 3m ago 24m 56.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (24m) 3m ago 24m 48.7M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:54:06.368 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (23m) 3m ago 23m 52.0M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (23m) 3m ago 23m 52.4M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (22m) 3m ago 22m 50.9M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (22m) 3m ago 22m 51.3M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (22m) 3m ago 22m 54.5M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (3m) 3m ago 21m 44.2M - 2.33.4 514e6a882f6e d11f0aa24855 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (19m) 3m ago 19m 74.7M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:54:06.369 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (19m) 3m ago 19m 77.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.34982 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: mgrmap e37: y(active, starting, since 0.996917s), standbys: x 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:54:06.589 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: Manager daemon y is now available 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:54:06.590 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.34982 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: mgrmap e37: y(active, starting, since 0.996917s), standbys: x 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:54:06.591 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:54:06.592 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:06 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:54:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.34982 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-21T12:54:06.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: mgrmap e37: y(active, starting, since 0.996917s), standbys: x 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:54:06.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:54:06.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:54:06.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: Manager daemon y is now available 2024-03-21T12:54:06.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:06.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:06.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:54:06.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:06.773 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:54:06.774 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:54:07.030 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:06.687+0000 7f4aa537d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:54:07.030 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:06.785+0000 7f4aa537d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:54:07.030 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:06.929+0000 7f4aa537d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:54:07.030 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.029+0000 7f4aa537d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr" 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T12:54:07.118 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T12:54:07.119 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:54:07.119 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:54:07.347 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.122+0000 7f4aa537d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:54:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: mgrmap e38: y(active, since 2s), standbys: x 2024-03-21T12:54:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:54:06] ENGINE Bus STARTING 2024-03-21T12:54:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:54:06] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:54:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:54:06] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:54:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: [21/Mar/2024:12:54:06] ENGINE Bus STARTED 2024-03-21T12:54:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3193476443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[24813]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: mgrmap e38: y(active, since 2s), standbys: x 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: [21/Mar/2024:12:54:06] ENGINE Bus STARTING 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: [21/Mar/2024:12:54:06] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: [21/Mar/2024:12:54:06] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: [21/Mar/2024:12:54:06] ENGINE Bus STARTED 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3193476443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:07 smithi172 ceph-mon[29218]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:07.606 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.403+0000 7f4aa537d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.518+0000 7f4aa537d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.605+0000 7f4aa537d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: mgrmap e38: y(active, since 2s), standbys: x 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: [21/Mar/2024:12:54:06] ENGINE Bus STARTING 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: [21/Mar/2024:12:54:06] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: [21/Mar/2024:12:54:06] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: [21/Mar/2024:12:54:06] ENGINE Bus STARTED 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3193476443' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:07.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-mon[25767]: from='client.15435 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:07.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.715+0000 7f4aa537d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:54:08.247 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:07.913+0000 7f4aa537d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:54:08.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:08 smithi177 ceph-mon[25767]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:08 smithi177 ceph-mon[25767]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-03-21T12:54:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:08 smithi177 ceph-mon[25767]: Cluster is now healthy 2024-03-21T12:54:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:08 smithi172 ceph-mon[24813]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:08 smithi172 ceph-mon[24813]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-03-21T12:54:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:08 smithi172 ceph-mon[24813]: Cluster is now healthy 2024-03-21T12:54:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:08 smithi172 ceph-mon[29218]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:08 smithi172 ceph-mon[29218]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 2 failed cephadm daemon(s)) 2024-03-21T12:54:08.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:08 smithi172 ceph-mon[29218]: Cluster is now healthy 2024-03-21T12:54:09.045 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:08.913+0000 7f4aa537d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:54:09.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:09 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:09.044+0000 7f4aa537d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:54:09.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:09 smithi177 ceph-mon[25767]: mgrmap e39: y(active, since 4s), standbys: x 2024-03-21T12:54:09.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:09 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:09.418+0000 7f4aa537d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:54:09.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:09 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:09.515+0000 7f4aa537d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:54:09.614 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:09 smithi172 ceph-mon[24813]: mgrmap e39: y(active, since 4s), standbys: x 2024-03-21T12:54:09.614 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:09 smithi172 ceph-mon[29218]: mgrmap e39: y(active, since 4s), standbys: x 2024-03-21T12:54:10.107 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:09 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:09.725+0000 7f4aa537d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:54:10.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:09 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:09.922+0000 7f4aa537d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:54:10.507 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:10.230+0000 7f4aa537d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[29218]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[29218]: mgrmap e40: y(active, since 6s), standbys: x 2024-03-21T12:54:10.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[24813]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:10.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:10 smithi172 ceph-mon[24813]: mgrmap e40: y(active, since 6s), standbys: x 2024-03-21T12:54:10.763 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:10.507+0000 7f4aa537d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:54:10.763 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:10.763+0000 7f4aa537d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:54:10.763 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-mon[25767]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:10.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:10.764 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:10 smithi177 ceph-mon[25767]: mgrmap e40: y(active, since 6s), standbys: x 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:11.602+0000 7f4aa537d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:11.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:11.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:54:11.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:11 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:54:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:11.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:11.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:11.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:54:11.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:11 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:54:12.537 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:12.206+0000 7f4aa537d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:54:12.538 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:12.313+0000 7f4aa537d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:12.537+0000 7f4aa537d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:54:12] ENGINE Bus STARTING 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: CherryPy Checker: 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: The Application mounted at '' has an empty config. 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:54:12] ENGINE Serving on http://:::9283 2024-03-21T12:54:12.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[45983]: [21/Mar/2024:12:54:12] ENGINE Bus STARTED 2024-03-21T12:54:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Standby manager daemon x restarted 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:54:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Standby manager daemon x restarted 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:54:13.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:54:13.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:54:13.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:13.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:54:13.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:13.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Standby manager daemon x restarted 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:54:13.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:13 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3980785726' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:54:13.945+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:14.124 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:14.125 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:54:14.126 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:14.127 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:14.128 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[24813]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T12:54:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[24813]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:54:14.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[24813]: mgrmap e41: y(active, since 8s), standbys: x 2024-03-21T12:54:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[29218]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T12:54:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[29218]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:54:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:14 smithi172 ceph-mon[29218]: mgrmap e41: y(active, since 8s), standbys: x 2024-03-21T12:54:14.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:14 smithi177 ceph-mon[25767]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T12:54:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:14 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:14 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:14 smithi177 ceph-mon[25767]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:54:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:14 smithi177 ceph-mon[25767]: mgrmap e41: y(active, since 8s), standbys: x 2024-03-21T12:54:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:15 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:15.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:15.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:15.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:15.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:15.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:15 smithi177 ceph-mon[25767]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:54:15.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:15 smithi177 ceph-mon[25767]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:54:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:15 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:15.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:54:15.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:15.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:15 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:15.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:15.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:15.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:15.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:15 smithi172 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:15 smithi172 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:15 smithi172 ceph-mon[29218]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T12:54:15.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:15 smithi172 ceph-mon[29218]: Reconfiguring daemon prometheus.a on smithi177 2024-03-21T12:54:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:16 smithi177 ceph-mon[25767]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T12:54:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:16 smithi172 ceph-mon[24813]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T12:54:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:16 smithi172 ceph-mon[29218]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-21T12:54:17.415 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:17.415 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 bash[60750]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.414Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T12:54:17.779 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[58067]: ts=2024-03-21T12:54:17.415Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T12:54:17.780 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 podman[60761]: 2024-03-21 12:54:17.425158638 +0000 UTC m=+0.116258744 container died d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:17.780 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 podman[60761]: 2024-03-21 12:54:17.779046484 +0000 UTC m=+0.470146583 container cleanup d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:17.780 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 bash[60761]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T12:54:18.055 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:17 smithi177 bash[60788]: Error: no container with name or ID "ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus.a" found: no such container 2024-03-21T12:54:18.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-mon[25767]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:54:18.314 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 podman[60776]: 2024-03-21 12:54:18.0553031 +0000 UTC m=+0.632551804 container remove d11f0aa248558e64391dd48e481db537b7823417044283944dc908194c442520 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:18.314 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T12:54:18.314 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 systemd[1]: Stopped Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:18.315 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.410s CPU time. 2024-03-21T12:54:18.315 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:18.315 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 podman[60863]: 2024-03-21 12:54:18.182226092 +0000 UTC m=+0.019238511 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-21T12:54:18.315 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 podman[60863]: 2024-03-21 12:54:18.3146787 +0000 UTC m=+0.151691162 container create 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:18.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:18 smithi172 ceph-mon[24813]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:54:18.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:18 smithi172 ceph-mon[29218]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:54:19.090 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 podman[60863]: 2024-03-21 12:54:18.626596021 +0000 UTC m=+0.463608445 container init 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:19.090 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 podman[60863]: 2024-03-21 12:54:18.629111593 +0000 UTC m=+0.466124010 container start 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:19.090 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.651Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.651Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.651Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.651Z caller=main.go:518 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.652Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.652Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.653Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.654Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.654Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.656Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.656Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.647µs 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.656Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 bash[60863]: 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:19.091 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:18.858Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-03-21T12:54:19.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.090Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-03-21T12:54:19.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.315Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-03-21T12:54:19.678 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.361Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-03-21T12:54:19.678 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.364Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-03-21T12:54:19.678 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.364Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-03-21T12:54:19.679 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.364Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=58.794µs wal_replay_duration=707.904992ms total_replay_duration=707.977223ms 2024-03-21T12:54:19.679 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.369Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:54:19.679 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.369Z caller=main.go:947 level=info msg="TSDB started" 2024-03-21T12:54:19.679 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.369Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:54:19.679 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.390Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.144049ms db_storage=632ns remote_storage=1.363µs web_handler=308ns query_engine=731ns scrape=606.376µs scrape_sd=101.858µs notify=12.535µs notify_sd=8.89µs rules=20.016599ms 2024-03-21T12:54:19.679 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:54:19.390Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-21T12:54:19.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-mon[25767]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 86 op/s 2024-03-21T12:54:19.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:19.972 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:19.972 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:54:19.972 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-mon[25767]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:54:19.972 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:19 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[29218]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 86 op/s 2024-03-21T12:54:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:20.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[29218]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[24813]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 86 op/s 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[24813]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:54:20.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:19 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:21.030 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:20 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:21.030 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:20 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:54:21.030 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:20 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:54:21.030 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:20 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:54:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:54:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:21.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:54:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:54:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:20 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:21.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:21 smithi177 ceph-mon[25767]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 86 op/s 2024-03-21T12:54:21.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:21.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:21 smithi177 ceph-mon[25767]: Upgrade: Updating mgr.x 2024-03-21T12:54:21.971 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:21 smithi177 ceph-mon[25767]: Deploying daemon mgr.x on smithi177 2024-03-21T12:54:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[24813]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 86 op/s 2024-03-21T12:54:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[24813]: Upgrade: Updating mgr.x 2024-03-21T12:54:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[24813]: Deploying daemon mgr.x on smithi177 2024-03-21T12:54:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[29218]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 86 op/s 2024-03-21T12:54:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[29218]: Upgrade: Updating mgr.x 2024-03-21T12:54:22.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:21 smithi172 ceph-mon[29218]: Deploying daemon mgr.x on smithi177 2024-03-21T12:54:22.507 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:22 smithi177 systemd[1]: Stopping Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:22.801 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:22 smithi177 podman[61215]: 2024-03-21 12:54:22.506728012 +0000 UTC m=+0.123278555 container died 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-21T12:54:23.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:22 smithi177 podman[61215]: 2024-03-21 12:54:22.884151609 +0000 UTC m=+0.500702197 container cleanup 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:54:23.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:22 smithi177 bash[61215]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x 2024-03-21T12:54:23.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 podman[61229]: 2024-03-21 12:54:23.035144499 +0000 UTC m=+0.528075314 container remove 41407098728c0ae24f02a67bb085fb22b28d050aa22fd9dbedf04581666a135f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T12:54:23.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:54:23.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-21T12:54:23.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 systemd[1]: Stopped Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:23.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service: Consumed 58.949s CPU time. 2024-03-21T12:54:23.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 systemd[1]: Starting Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:23.946 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:23 smithi177 ceph-mon[25767]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-21T12:54:23.946 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 podman[61362]: 2024-03-21 12:54:23.560560689 +0000 UTC m=+0.018178432 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:54:23.946 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 podman[61362]: 2024-03-21 12:54:23.679149398 +0000 UTC m=+0.136767168 container create c7a18aa3e6205f001da96c22b6e045db187aa359d298ef6929db7fbfd2b7c8fd (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-21T12:54:23.946 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 podman[61362]: 2024-03-21 12:54:23.871975891 +0000 UTC m=+0.329593632 container init c7a18aa3e6205f001da96c22b6e045db187aa359d298ef6929db7fbfd2b7c8fd (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-03-21T12:54:23.947 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 podman[61362]: 2024-03-21 12:54:23.874427888 +0000 UTC m=+0.332045619 container start c7a18aa3e6205f001da96c22b6e045db187aa359d298ef6929db7fbfd2b7c8fd (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T12:54:23.947 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 ceph-mgr[61380]: -- 172.21.15.177:0/417579797 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5625a310a5a0 con 0x5625a30de800 2024-03-21T12:54:24.109 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:23 smithi172 ceph-mon[24813]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-21T12:54:24.109 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:23 smithi172 ceph-mon[29218]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 99 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-21T12:54:24.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 bash[61362]: c7a18aa3e6205f001da96c22b6e045db187aa359d298ef6929db7fbfd2b7c8fd 2024-03-21T12:54:24.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:23 smithi177 systemd[1]: Started Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:24.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:24.440+0000 7fa318183200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:54:24.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:24 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:24.543+0000 7fa318183200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:54:25.237 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:25.237 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:25.237 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:25.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:25.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:25.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:25.495 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:25.236+0000 7fa318183200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:54:25.495 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:25.333+0000 7fa318183200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:54:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:54:25] "GET /metrics HTTP/1.1" 200 38000 "" "Prometheus/2.33.4" 2024-03-21T12:54:25.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:25.494+0000 7fa318183200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:54:25.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:25.591+0000 7fa318183200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:54:25.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:25.679+0000 7fa318183200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:54:26.266 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:26 smithi177 ceph-mon[25767]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T12:54:26.267 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:25 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:25.952+0000 7fa318183200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:54:26.267 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:26 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:26.062+0000 7fa318183200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:54:26.267 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:26 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:26.155+0000 7fa318183200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:54:26.267 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:26 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:26.266+0000 7fa318183200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:54:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:26 smithi172 ceph-mon[24813]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T12:54:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:26 smithi172 ceph-mon[29218]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T12:54:26.570 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:26 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:26.457+0000 7fa318183200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:54:27.597 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:27.462+0000 7fa318183200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:54:27.597 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:27.597+0000 7fa318183200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:54:28.271 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:28 smithi177 ceph-mon[25767]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T12:54:28.271 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:27 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:27.957+0000 7fa318183200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:54:28.272 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:28.057+0000 7fa318183200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:54:28.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:28 smithi172 ceph-mon[24813]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T12:54:28.423 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:28 smithi172 ceph-mon[29218]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-21T12:54:28.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:28.271+0000 7fa318183200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:54:28.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:28.469+0000 7fa318183200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:54:29.052 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:28 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:28.774+0000 7fa318183200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:54:29.309 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:29 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:29.051+0000 7fa318183200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:54:29.309 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:29 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:29.308+0000 7fa318183200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:54:29.309 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:29 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:29.309 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:29 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:29 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:29 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:29 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:29.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:29 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.104 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:30 smithi177 ceph-mon[25767]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T12:54:30.105 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:30 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.105 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:30 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:30 smithi172 ceph-mon[24813]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T12:54:30.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:30 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.323 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:30 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.323 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:30 smithi172 ceph-mon[29218]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-21T12:54:30.323 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:30 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.323 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:30 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:30.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:30.168+0000 7fa318183200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:54:31.071 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:30.748+0000 7fa318183200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:54:31.071 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:30 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:30.858+0000 7fa318183200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:54:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:31.087+0000 7fa318183200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:54:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: [21/Mar/2024:12:54:31] ENGINE Bus STARTING 2024-03-21T12:54:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: CherryPy Checker: 2024-03-21T12:54:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: The Application mounted at '' has an empty config. 2024-03-21T12:54:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:54:31.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: [21/Mar/2024:12:54:31] ENGINE Serving on http://:::9283 2024-03-21T12:54:31.359 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:54:31 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: [21/Mar/2024:12:54:31] ENGINE Bus STARTED 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: Standby manager daemon x restarted 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: Standby manager daemon x started 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:54:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:32 smithi177 ceph-mon[25767]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: Standby manager daemon x restarted 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: Standby manager daemon x started 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:54:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[24813]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: Standby manager daemon x restarted 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:54:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:32 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/3169357426' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: mgrmap e42: y(active, since 27s), standbys: x 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:33 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: mgrmap e42: y(active, since 27s), standbys: x 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: mgrmap e42: y(active, since 27s), standbys: x 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[24813]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[24813]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[24813]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[29218]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[29218]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[29218]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:34.357 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:34 smithi177 ceph-mon[25767]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-21T12:54:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:34 smithi177 ceph-mon[25767]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 990 B/s rd, 0 op/s 2024-03-21T12:54:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:34 smithi177 ceph-mon[25767]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:34 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:34 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:34.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:54:34.444+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:54:34.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:34.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:34.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:34.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:34.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:34 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:54:35] "GET /metrics HTTP/1.1" 200 38000 "" "Prometheus/2.33.4" 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-21T12:54:35.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-21T12:54:35.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-21T12:54:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:36.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:54:36.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:54:36.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all mgr 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all crash 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all mds 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:54:36.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:54:36.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all nfs 2024-03-21T12:54:36.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:54:36.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:36.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:36.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:36.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:36.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:36.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:36.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:36.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:36.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:36.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:36.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:36.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:36.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:36.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all mgr 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all crash 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all mds 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:54:36.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all nfs 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all mgr 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all crash 2024-03-21T12:54:36.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all mds 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all nfs 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:36.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:36.896 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:36 smithi172 systemd[1]: Stopping Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:37 smithi177 ceph-mon[25767]: Upgrade: Updating node-exporter.a (1/2) 2024-03-21T12:54:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:37 smithi177 ceph-mon[25767]: Deploying daemon node-exporter.a on smithi172 2024-03-21T12:54:37.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:37 smithi172 ceph-mon[24813]: Upgrade: Updating node-exporter.a (1/2) 2024-03-21T12:54:37.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:37 smithi172 ceph-mon[24813]: Deploying daemon node-exporter.a on smithi172 2024-03-21T12:54:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:37 smithi172 ceph-mon[29218]: Upgrade: Updating node-exporter.a (1/2) 2024-03-21T12:54:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:37 smithi172 ceph-mon[29218]: Deploying daemon node-exporter.a on smithi172 2024-03-21T12:54:37.396 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 podman[75889]: 2024-03-21 12:54:37.079335349 +0000 UTC m=+0.118184417 container died 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:37.468 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:54:37.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 podman[75889]: 2024-03-21 12:54:37.502829133 +0000 UTC m=+0.541678209 container cleanup 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:37.646 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 bash[75889]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T12:54:38.008 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 podman[75903]: 2024-03-21 12:54:37.745542523 +0000 UTC m=+0.666029077 container remove 1145f3da8c8f88e16ca17ca0aa9bf293752eaa8150a0f0bd569509ce3872cc3a (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:38.009 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:54:38.009 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T12:54:38.009 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 systemd[1]: Stopped Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:38.009 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:37 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 5.148s CPU time. 2024-03-21T12:54:38.174 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:54:38.174 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (13m) 28s ago 21m 24.5M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:54:38.174 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 10s ago 20m - - 2024-03-21T12:54:38.174 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 error 28s ago 20m 62.3M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (14s) 10s ago 25m 65.6M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (10m) 28s ago 27m 474M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (27m) 28s ago 27m 71.3M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (25m) 10s ago 25m 55.9M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (26m) 28s ago 26m 55.9M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (13m) 28s ago 21m 19.0M - 1.3.1 1dbe0e931976 1145f3da8c8f 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (12m) 10s ago 21m 17.5M - 1.3.1 1dbe0e931976 4437b302e23d 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (25m) 28s ago 25m 56.2M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (24m) 28s ago 24m 56.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (24m) 28s ago 24m 48.7M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (24m) 28s ago 24m 52.0M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (23m) 10s ago 23m 54.9M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (23m) 10s ago 23m 51.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:54:38.175 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (23m) 10s ago 23m 52.6M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:54:38.176 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (22m) 10s ago 22m 55.7M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:54:38.176 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (19s) 10s ago 21m 56.4M - 2.33.4 514e6a882f6e 5ae5220ab590 2024-03-21T12:54:38.176 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (20m) 28s ago 20m 75.5M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:54:38.176 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (20m) 10s ago 20m 78.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:54:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:38 smithi177 ceph-mon[25767]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:38.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:38 smithi172 ceph-mon[24813]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:38.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:38 smithi172 ceph-mon[29218]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:38.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:38 smithi172 systemd[1]: Starting Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:38.602 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:54:38.602 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:54:38.602 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:54:38.602 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:38.602 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:54:38.602 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:54:38.603 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:54:38.896 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:38 smithi172 bash[76127]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-21T12:54:38.931 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:54:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T12:54:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:54:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T12:54:38.931 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T12:54:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr" 2024-03-21T12:54:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:54:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T12:54:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-21T12:54:38.932 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:54:38.932 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:54:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:39 smithi177 ceph-mon[25767]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:39 smithi177 ceph-mon[25767]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:39 smithi177 ceph-mon[25767]: from='client.25360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:39 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2401066219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[24813]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[24813]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[24813]: from='client.25360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2401066219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[29218]: from='client.25348 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[29218]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[29218]: from='client.25360 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:39.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:39 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2401066219' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:39.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:38 smithi172 bash[76127]: Getting image source signatures 2024-03-21T12:54:39.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:38 smithi172 bash[76127]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-21T12:54:39.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:38 smithi172 bash[76127]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T12:54:39.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:38 smithi172 bash[76127]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-21T12:54:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:40 smithi177 ceph-mon[25767]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:40 smithi177 ceph-mon[25767]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:40 smithi172 ceph-mon[29218]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:40 smithi172 ceph-mon[29218]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:40.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:40 smithi172 ceph-mon[24813]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:40.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:40 smithi172 ceph-mon[24813]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:54:40.396 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:40 smithi172 bash[76127]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T12:54:40.731 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:40 smithi172 bash[76127]: Writing manifest to image destination 2024-03-21T12:54:41.058 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:40 smithi172 podman[76127]: 2024-03-21 12:54:40.708525354 +0000 UTC m=+2.400162427 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T12:54:41.058 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:40 smithi172 podman[76127]: 2024-03-21 12:54:40.806910763 +0000 UTC m=+2.498547823 container create ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:41.396 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 podman[76127]: 2024-03-21 12:54:41.058688895 +0000 UTC m=+2.750325965 container init ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:41.396 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 podman[76127]: 2024-03-21 12:54:41.062452372 +0000 UTC m=+2.754089432 container start ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.066Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.066Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.067Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.067Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.067Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T12:54:41.397 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T12:54:41.398 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T12:54:41.398 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T12:54:41.399 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T12:54:41.400 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T12:54:41.401 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T12:54:41.402 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a[76204]: ts=2024-03-21T12:54:41.068Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T12:54:41.403 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 bash[76127]: ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 2024-03-21T12:54:41.403 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 12:54:41 smithi172 systemd[1]: Started Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:42 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:42 smithi177 ceph-mon[25767]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-21T12:54:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:42 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:42 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[24813]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-21T12:54:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[29218]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-21T12:54:42.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:42.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:42 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:43.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:43 smithi177 systemd[1]: Stopping Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:43.966 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:43 smithi177 podman[62715]: 2024-03-21 12:54:43.628144055 +0000 UTC m=+0.114924712 container died 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:54:43.967 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:43 smithi177 podman[62715]: 2024-03-21 12:54:43.967475068 +0000 UTC m=+0.454255724 container cleanup 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:54:44.293 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:44 smithi177 ceph-mon[25767]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:54:44.293 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:44 smithi177 ceph-mon[25767]: Upgrade: Updating node-exporter.b (2/2) 2024-03-21T12:54:44.293 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:44 smithi177 ceph-mon[25767]: Deploying daemon node-exporter.b on smithi177 2024-03-21T12:54:44.294 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:43 smithi177 bash[62715]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T12:54:44.294 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 podman[62729]: 2024-03-21 12:54:44.193697277 +0000 UTC m=+0.567862247 container remove 4437b302e23ddacffd20571986ae483c47029958ac0a30a09c40663716d1b806 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:54:44.294 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T12:54:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:44 smithi172 ceph-mon[24813]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:54:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:44 smithi172 ceph-mon[24813]: Upgrade: Updating node-exporter.b (2/2) 2024-03-21T12:54:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:44 smithi172 ceph-mon[24813]: Deploying daemon node-exporter.b on smithi177 2024-03-21T12:54:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:44 smithi172 ceph-mon[29218]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:54:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:44 smithi172 ceph-mon[29218]: Upgrade: Updating node-exporter.b (2/2) 2024-03-21T12:54:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:44 smithi172 ceph-mon[29218]: Deploying daemon node-exporter.b on smithi177 2024-03-21T12:54:44.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T12:54:44.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 systemd[1]: Stopped Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:44.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 5.020s CPU time. 2024-03-21T12:54:44.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 systemd[1]: Starting Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:54:45.107 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:44 smithi177 bash[62864]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-21T12:54:45.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:45 smithi177 bash[62864]: Getting image source signatures 2024-03-21T12:54:45.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:45 smithi177 bash[62864]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-21T12:54:45.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:45 smithi177 bash[62864]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-21T12:54:45.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:45 smithi177 bash[62864]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-21T12:54:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:54:45] "GET /metrics HTTP/1.1" 200 38001 "" "Prometheus/2.33.4" 2024-03-21T12:54:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:46 smithi172 ceph-mon[24813]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:46 smithi172 ceph-mon[29218]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:46.466 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:46 smithi177 ceph-mon[25767]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:46.858 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:46 smithi177 bash[62864]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-21T12:54:46.858 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:46 smithi177 bash[62864]: Writing manifest to image destination 2024-03-21T12:54:47.215 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:46 smithi177 podman[62864]: 2024-03-21 12:54:46.872801394 +0000 UTC m=+2.268164383 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-21T12:54:47.215 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:46 smithi177 podman[62864]: 2024-03-21 12:54:46.988513049 +0000 UTC m=+2.383876040 container create d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:54:47.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 podman[62864]: 2024-03-21 12:54:47.215583402 +0000 UTC m=+2.610946406 container init d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:54:47.608 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 podman[62864]: 2024-03-21 12:54:47.218689789 +0000 UTC m=+2.614052786 container start d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.222Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.222Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.223Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.223Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T12:54:47.609 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T12:54:47.610 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T12:54:47.611 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T12:54:47.612 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T12:54:47.613 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.224Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T12:54:47.613 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.225Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-21T12:54:47.613 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b[62915]: ts=2024-03-21T12:54:47.225Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T12:54:47.613 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 bash[62864]: d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 2024-03-21T12:54:47.613 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 12:54:47 smithi177 systemd[1]: Started Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:54:48.376 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:48 smithi177 ceph-mon[25767]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:48.377 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:48.377 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:48.377 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[24813]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[29218]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:48.381 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:50.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:50 smithi177 ceph-mon[25767]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:50 smithi172 ceph-mon[24813]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:50 smithi172 ceph-mon[29218]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:51.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:51.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[29218]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[24813]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:52 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:52 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:54:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:52 smithi177 ceph-mon[25767]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:52 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:52 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:52 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:52.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:52 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.580 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.580 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.580 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[24813]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:53.580 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.581 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.581 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.581 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.581 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[29218]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:53.581 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.581 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:53 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:53 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:53 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:53 smithi177 ceph-mon[25767]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:54:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:53 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:53.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:53 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:55.596 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:54:55] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.33.4" 2024-03-21T12:54:55.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:55 smithi177 ceph-mon[25767]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:55.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:55 smithi172 ceph-mon[24813]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:55.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:55 smithi172 ceph-mon[29218]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:54:56.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:56.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:56 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:54:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:56 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:54:58.029 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[24813]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-21T12:54:58.029 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[24813]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-03-21T12:54:58.030 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:58.030 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:58.030 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[29218]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-21T12:54:58.030 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[29218]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-03-21T12:54:58.030 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:58.030 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:57 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:57 smithi177 ceph-mon[25767]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-21T12:54:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:57 smithi177 ceph-mon[25767]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-03-21T12:54:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:57 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:54:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:57 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:54:58.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:54:58.028+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:58.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:54:58.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.946 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:58.947 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:58.948 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.949 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:58.950 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:58.951 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:58 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:59.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:54:59.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:59.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:59.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:59.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:54:59.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:54:59.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:59.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:54:59.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:00.049 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:54:59 smithi172 ceph-mon[24813]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:00.049 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:54:59 smithi172 ceph-mon[29218]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:00.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:54:59 smithi177 ceph-mon[25767]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:00.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:00 smithi177 ceph-mon[25767]: Upgrade: Updating prometheus.a 2024-03-21T12:55:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:00 smithi177 ceph-mon[25767]: Deploying daemon prometheus.a on smithi177 2024-03-21T12:55:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:00 smithi172 ceph-mon[24813]: Upgrade: Updating prometheus.a 2024-03-21T12:55:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:00 smithi172 ceph-mon[24813]: Deploying daemon prometheus.a on smithi177 2024-03-21T12:55:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:00 smithi172 ceph-mon[29218]: Upgrade: Updating prometheus.a 2024-03-21T12:55:01.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:00 smithi172 ceph-mon[29218]: Deploying daemon prometheus.a on smithi177 2024-03-21T12:55:02.044 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:01 smithi177 ceph-mon[25767]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:02.045 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:01 smithi172 ceph-mon[24813]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:01 smithi172 ceph-mon[29218]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:03.395 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:55:03.694 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.394Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:55:03.694 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-21T12:55:03.694 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-21T12:55:03.694 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-21T12:55:03.694 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[60877]: ts=2024-03-21T12:55:03.395Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-21T12:55:03.695 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 podman[64233]: 2024-03-21 12:55:03.405044621 +0000 UTC m=+0.118965220 container died 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:04.086 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:03 smithi177 ceph-mon[25767]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:04.086 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 podman[64233]: 2024-03-21 12:55:03.693882431 +0000 UTC m=+0.407803044 container cleanup 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:04.086 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 bash[64233]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T12:55:04.087 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:03 smithi177 podman[64247]: 2024-03-21 12:55:03.895520909 +0000 UTC m=+0.492745984 container remove 5ae5220ab5905a3f09fdbe65ff924b547f5cf0a6d12e1b2c36b407b3a3d1f0af (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:03 smithi172 ceph-mon[24813]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:03 smithi172 ceph-mon[29218]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:04.366 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T12:55:04.366 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 systemd[1]: Stopped Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:55:04.366 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.265s CPU time. 2024-03-21T12:55:04.366 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 systemd[1]: Starting Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:55:04.782 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 podman[64377]: 2024-03-21 12:55:04.425756217 +0000 UTC m=+0.018847883 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 podman[64377]: 2024-03-21 12:55:04.52965905 +0000 UTC m=+0.122750766 container create b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 podman[64377]: 2024-03-21 12:55:04.740047841 +0000 UTC m=+0.333139509 container init b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 podman[64377]: 2024-03-21 12:55:04.742516316 +0000 UTC m=+0.335607979 container start b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.765Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.765Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.765Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi177 (none))" 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.765Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.765Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.767Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.767Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.768Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.768Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T12:55:04.783 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.770Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T12:55:04.784 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.771Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.658µs 2024-03-21T12:55:04.784 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.771Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T12:55:04.784 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.778Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.791Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.801Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.804Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.807Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.808Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.808Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.808Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.924µs wal_replay_duration=37.49953ms wbl_replay_duration=133ns total_replay_duration=37.565272ms 2024-03-21T12:55:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.812Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.812Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.812Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.832Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.430895ms db_storage=806ns remote_storage=1.175µs web_handler=459ns query_engine=744ns scrape=445.565µs scrape_sd=94.787µs notify=17.269µs notify_sd=14.651µs rules=19.535166ms tracing=5.925µs 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.832Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T12:55:04.832Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 bash[64377]: b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a 2024-03-21T12:55:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 12:55:04 smithi177 systemd[1]: Started Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:55:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:05 smithi177 ceph-mon[25767]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:05 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:05 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:05 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:05 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[24813]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[29218]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:55:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:05 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:08.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:07 smithi172 ceph-mon[24813]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:08.163 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:07 smithi172 ceph-mon[29218]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:08.220 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:07 smithi177 ceph-mon[25767]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:09.293 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:55:09.936 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:55:09.936 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (13m) 18s ago 21m 24.6M - 0.23.0 ba2b418f427c b12c24cd5ca4 2024-03-21T12:55:09.936 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 error 1s ago 21m - - 2024-03-21T12:55:09.936 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 18s ago 20m 62.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:55:09.936 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (46s) 1s ago 26m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (10m) 18s ago 27m 501M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (27m) 18s ago 27m 75.4M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (26m) 1s ago 26m 57.2M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (26m) 18s ago 26m 59.3M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (29s) 18s ago 22m 7968k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (23s) 1s ago 22m 7549k - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (25m) 18s ago 25m 56.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (25m) 18s ago 25m 57.5M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (25m) 18s ago 25m 48.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (24m) 18s ago 24m 52.1M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (24m) 1s ago 24m 55.0M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (23m) 1s ago 23m 51.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (23m) 1s ago 23m 52.7M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (23m) 1s ago 23m 55.7M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:55:09.937 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (5s) 1s ago 22m 46.9M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:55:09.938 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (20m) 18s ago 20m 75.9M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:55:09.938 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (20m) 1s ago 20m 78.4M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:55:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:09 smithi177 ceph-mon[25767]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:09 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:09 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:09 smithi172 ceph-mon[24813]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:09 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:09 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:09 smithi172 ceph-mon[29218]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:09 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:10.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:09 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:10.330 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:55:10.331 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:55:10.332 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:55:10.332 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:55:10.332 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr" 2024-03-21T12:55:10.660 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:55:10.661 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T12:55:10.661 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-21T12:55:10.661 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:55:10.661 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:55:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:10 smithi177 ceph-mon[25767]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:10 smithi177 ceph-mon[25767]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:10 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:10 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:10 smithi177 ceph-mon[25767]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:10 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/2229747054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[24813]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[24813]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[24813]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2229747054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[29218]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[29218]: from='client.25384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[29218]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:10 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2229747054' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:12.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:11 smithi177 ceph-mon[25767]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:12.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:12.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:11 smithi177 ceph-mon[25767]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:11 smithi172 ceph-mon[24813]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:11 smithi172 ceph-mon[24813]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:11 smithi172 ceph-mon[29218]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:11 smithi172 ceph-mon[29218]: from='client.25402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:14.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:14.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:55:14.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:14.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:14.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:14 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:55:14.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:14.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:14.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:14.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:14 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:15.134 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:15.134 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:15.135 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:15.135 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:15.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:55:15.133+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:15.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:15.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:15.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:15.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:15.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:15.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:15.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:15.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:15.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:15.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:15.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:15.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:55:15] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-21T12:55:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:15 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:15 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:16.379 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:16.379 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:16.379 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:16.379 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:16.379 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:16.379 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:16.380 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:16.381 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.382 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:16.383 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:16.384 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:16.385 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:16.386 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:16.387 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.388 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:16.389 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.391 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.391 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:16 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:16.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:55:16.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:16.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:16.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:16 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:17 smithi172 ceph-mon[24813]: Upgrade: Updating alertmanager.a 2024-03-21T12:55:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:17 smithi172 ceph-mon[24813]: Deploying daemon alertmanager.a on smithi172 2024-03-21T12:55:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:17 smithi172 ceph-mon[24813]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:17 smithi172 ceph-mon[29218]: Upgrade: Updating alertmanager.a 2024-03-21T12:55:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:17 smithi172 ceph-mon[29218]: Deploying daemon alertmanager.a on smithi172 2024-03-21T12:55:17.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:17 smithi172 ceph-mon[29218]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:17 smithi177 ceph-mon[25767]: Upgrade: Updating alertmanager.a 2024-03-21T12:55:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:17 smithi177 ceph-mon[25767]: Deploying daemon alertmanager.a on smithi172 2024-03-21T12:55:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:17 smithi177 ceph-mon[25767]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:20.382 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:20 smithi172 ceph-mon[24813]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:20.382 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:20 smithi172 systemd[1]: Stopping Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:55:20.382 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:20 smithi172 ceph-mon[29218]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:20 smithi177 ceph-mon[25767]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:55:20.646 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[58913]: level=info ts=2024-03-21T12:55:20.382Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-21T12:55:20.646 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:20 smithi172 podman[78068]: 2024-03-21 12:55:20.433518718 +0000 UTC m=+0.170280041 container died b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:21.066 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:20 smithi172 podman[78068]: 2024-03-21 12:55:20.816027944 +0000 UTC m=+0.552789272 container cleanup b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:21.066 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:20 smithi172 bash[78068]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T12:55:21.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:21 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:21.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:21 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:21.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 podman[78082]: 2024-03-21 12:55:21.067146716 +0000 UTC m=+0.630493131 container remove b12c24cd5ca4a1927998620aeee6fbce730ba9816095b23c4f9d34433af07a2c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:21.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 podman[78082]: 2024-03-21 12:55:21.284645456 +0000 UTC m=+0.847991869 volume remove 92fa5842a4ba66a13d4983dc452870d4a108214df43c4ac25103e9d0c8002f21 2024-03-21T12:55:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:21 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:21.824 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T12:55:21.824 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 systemd[1]: Stopped Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:55:21.824 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.536s CPU time. 2024-03-21T12:55:21.824 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 systemd[1]: Starting Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:55:22.128 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 podman[78209]: 2024-03-21 12:55:21.726987068 +0000 UTC m=+0.018272248 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-21T12:55:22.129 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 podman[78209]: 2024-03-21 12:55:21.827328433 +0000 UTC m=+0.118613628 volume create d62446a4359cd26f5f8d6102d53aacb8baec97b98e4c4bc081063c865950408e 2024-03-21T12:55:22.129 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:21 smithi172 podman[78209]: 2024-03-21 12:55:21.927814025 +0000 UTC m=+0.219099180 container create 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-mon[24813]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-21T12:55:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-mon[29218]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 podman[78209]: 2024-03-21 12:55:22.230611829 +0000 UTC m=+0.521896996 container init 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 podman[78209]: 2024-03-21 12:55:22.233380909 +0000 UTC m=+0.524666079 container start 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.244Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.244Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.246Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.172 port=9094 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.247Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T12:55:22.397 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.271Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T12:55:22.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.271Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T12:55:22.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.274Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-21T12:55:22.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:22.274Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T12:55:22.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 bash[78209]: 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f 2024-03-21T12:55:22.398 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:22 smithi172 systemd[1]: Started Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:55:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:22 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:22.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:22 smithi177 ceph-mon[25767]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[24813]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:23.470 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:23.471 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:23 smithi172 ceph-mon[29218]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:23.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:23 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:23.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:23 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:23.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:23 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:23.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:23 smithi177 ceph-mon[25767]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:24.646 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:24.248Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000366994s 2024-03-21T12:55:25.569 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:55:25] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.43.0" 2024-03-21T12:55:26.455 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:26 smithi172 ceph-mon[24813]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:26.455 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:26 smithi172 ceph-mon[29218]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:26.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:26 smithi177 ceph-mon[25767]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:27 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:27 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:27 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:27 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:27.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:27 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:28.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:28 smithi172 ceph-mon[24813]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:28.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:28.448 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:28.448 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:28 smithi172 ceph-mon[29218]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:28.449 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:28.449 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:28 smithi177 ceph-mon[25767]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:28 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:28 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:29.444 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:29 smithi172 ceph-mon[24813]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:29.444 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:29 smithi172 ceph-mon[29218]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:29.509 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:29 smithi177 ceph-mon[25767]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:32.396 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T12:55:32.250Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002200218s 2024-03-21T12:55:32.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:32.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:32.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:32.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:32 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:32.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:32.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:32.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:32 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[24813]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-21T12:55:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[24813]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-21T12:55:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:33.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[29218]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-21T12:55:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[29218]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-21T12:55:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:33.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:55:33.276+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:33.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:33.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:33.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:33.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:33.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:33.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:33 smithi177 ceph-mon[25767]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-21T12:55:33.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:33 smithi177 ceph-mon[25767]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-21T12:55:33.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:33 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:33.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:33 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:34 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:34.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:35.537 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:35 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:35.537 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:35.538 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.539 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.540 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:35.541 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:35.542 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:35 smithi177 ceph-mon[25767]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:35 smithi177 ceph-mon[25767]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:35 smithi177 ceph-mon[25767]: Upgrade: Updating grafana.a 2024-03-21T12:55:35.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:35 smithi177 ceph-mon[25767]: Deploying daemon grafana.a on smithi177 2024-03-21T12:55:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:55:35] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.43.0" 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:35.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:35.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:35.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:35.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:35.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:35.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:35.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:35.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[24813]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[24813]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[24813]: Upgrade: Updating grafana.a 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[24813]: Deploying daemon grafana.a on smithi177 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:35.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:35.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[29218]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T12:55:35.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[29218]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:35.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[29218]: Upgrade: Updating grafana.a 2024-03-21T12:55:35.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:35 smithi172 ceph-mon[29218]: Deploying daemon grafana.a on smithi177 2024-03-21T12:55:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:36 smithi177 ceph-mon[25767]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-21T12:55:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:36 smithi172 ceph-mon[24813]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-21T12:55:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:36 smithi172 ceph-mon[29218]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 625 B/s rd, 0 op/s 2024-03-21T12:55:36.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:38.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:37 smithi177 systemd[1]: Starting Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:55:38.443 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 podman[66341]: 2024-03-21 12:55:38.095950726 +0000 UTC m=+0.018997071 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-21T12:55:38.443 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 podman[66341]: 2024-03-21 12:55:38.218773444 +0000 UTC m=+0.141819759 container create 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d (image=quay.io/ceph/grafana:9.4.12, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T12:55:38.443 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 podman[66341]: 2024-03-21 12:55:38.43748354 +0000 UTC m=+0.360529877 container init 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d (image=quay.io/ceph/grafana:9.4.12, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T12:55:38.443 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 podman[66341]: 2024-03-21 12:55:38.44322252 +0000 UTC m=+0.366268835 container start 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d (image=quay.io/ceph/grafana:9.4.12, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T12:55:38.699 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 bash[66341]: 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d 2024-03-21T12:55:38.699 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 systemd[1]: Started Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588099306Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588327425Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588340126Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588345216Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588350598Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588354916Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588358906Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588363151Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588368029Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588373058Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588378044Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588382247Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588386596Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T12:55:38.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588391568Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588395795Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588399706Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588403842Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588408099Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=settings t=2024-03-21T12:55:38.588412574Z level=info msg="App mode production" 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=sqlstore t=2024-03-21T12:55:38.588480388Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=sqlstore t=2024-03-21T12:55:38.588495756Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.600234289Z level=info msg="Starting DB migrations" 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.603802872Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.615279475Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.616422085Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.619056355Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-21T12:55:38.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.621823693Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.624480452Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.627113184Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.629861218Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.632509164Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.63511461Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.63782236Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.640490901Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.643164796Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-21T12:55:38.702 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.644240106Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.64696958Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.649515212Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.652148012Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.653205702Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.65583807Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.658552904Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.661128485Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.662256834Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.664690781Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.667308341Z level=info msg="Executing migration" id="create alert_image table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.670033264Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.670545302Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.673131838Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-21T12:55:38.703 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.673602251Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.675671608Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.675764929Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.676072259Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.676668991Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.677213987Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.678152275Z level=info msg="Executing migration" id="create secrets table" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.678808425Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.685076686Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.686206866Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.687007111Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.693311967Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-21T12:55:38.704 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.699646301Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.701294759Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.702081823Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.702857012Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.703665471Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.704239954Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.70601542Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.706790058Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.707603138Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.707956323Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.708827189Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.709672882Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-21T12:55:39.109 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.71082503Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.711148512Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.712017034Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.712823965Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.713693779Z level=info msg="Executing migration" id="add correlation config column" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.714985504Z level=info msg="Executing migration" id="create entity_events table" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.715819965Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.716627138Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T12:55:39.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-mon[25767]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:39.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:39.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:39.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:39.111 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.716707979Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T12:55:39.111 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.717481158Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T12:55:39.111 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.717560609Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T12:55:39.111 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.717841619Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.718703791Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.71952164Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.720305143Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.720745267Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.723369501Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.726175463Z level=info msg="Executing migration" id="Drop public config table" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.728921749Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.729377176Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.73202719Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.732485315Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.73480955Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.738724792Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-21T12:55:39.112 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.739936137Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.742589345Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.745329623Z level=info msg="Executing migration" id="add share column" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.747814369Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.748313566Z level=info msg="Executing migration" id="create file table" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.750964601Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.753709759Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.754202565Z level=info msg="Executing migration" id="create file_meta table" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.754989721Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.755829215Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.756150018Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.757285968Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.758095846Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-21T12:55:39.113 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.758649954Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.759979886Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.760737719Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.761560592Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.761969271Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.762833742Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.763697777Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.764947453Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.766125351Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.766980887Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.780177437Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.78100154Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.781475896Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.785890217Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-21T12:55:39.114 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.786765742Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.787500352Z level=info msg="Executing migration" id="create folder table" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.787901019Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.788720795Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.789566127Z level=info msg="Executing migration" id="Update folder title length" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.790419079Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=migrator t=2024-03-21T12:55:38.790894908Z level=info msg="migrations completed" performed=105 skipped=376 duration=187.132434ms 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=sqlstore t=2024-03-21T12:55:38.79121821Z level=info msg="Created default organization" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=plugin.finder t=2024-03-21T12:55:38.8232434Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=secrets t=2024-03-21T12:55:38.823366431Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=query_data t=2024-03-21T12:55:38.827342047Z level=info msg="Query Service initialization" 2024-03-21T12:55:39.115 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=live.push_http t=2024-03-21T12:55:38.829700819Z level=info msg="Live Push Gateway initialization" 2024-03-21T12:55:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[29218]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:39.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[24813]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:39.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:39.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:39.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:38 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=infra.usagestats.collector t=2024-03-21T12:55:39.465097098Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=provisioning.datasources t=2024-03-21T12:55:39.467222891Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=provisioning.datasources t=2024-03-21T12:55:39.467231075Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=provisioning.datasources t=2024-03-21T12:55:39.467326437Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=provisioning.datasources t=2024-03-21T12:55:39.478541574Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=provisioning.alerting t=2024-03-21T12:55:39.489222751Z level=info msg="starting to provision alerting" 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=provisioning.alerting t=2024-03-21T12:55:39.489233975Z level=info msg="finished to provision alerting" 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=grafanaStorageLogger t=2024-03-21T12:55:39.489358366Z level=info msg="storage starting" 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=ngalert.state.manager t=2024-03-21T12:55:39.489531865Z level=info msg="Warming state cache for startup" 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=http.server t=2024-03-21T12:55:39.492163561Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T12:55:39.700 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=ngalert.state.manager t=2024-03-21T12:55:39.502907648Z level=info msg="State cache has been initialized" states=0 duration=13.374455ms 2024-03-21T12:55:39.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=ticker t=2024-03-21T12:55:39.50296179Z level=info msg=starting first_tick=2024-03-21T12:55:40Z 2024-03-21T12:55:39.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=ngalert.multiorg.alertmanager t=2024-03-21T12:55:39.502970852Z level=info msg="starting MultiOrg Alertmanager" 2024-03-21T12:55:39.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=sqlstore.transactions t=2024-03-21T12:55:39.513343768Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T12:55:39.701 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 12:55:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=sqlstore.transactions t=2024-03-21T12:55:39.524341647Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-21T12:55:41.021 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:55:41.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:40 smithi172 ceph-mon[24813]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:41.022 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:40 smithi172 ceph-mon[29218]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:41.054 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:40 smithi177 ceph-mon[25767]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (19s) 14s ago 22m 16.7M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 starting - - - - 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 14s ago 21m 64.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (78s) 15s ago 26m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (11m) 14s ago 28m 504M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (28m) 14s ago 28m 80.8M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (26m) 15s ago 26m 60.9M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (27m) 14s ago 27m 63.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (60s) 14s ago 22m 8354k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (54s) 15s ago 22m 12.6M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:55:41.663 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (26m) 14s ago 26m 56.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (26m) 14s ago 26m 57.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (25m) 14s ago 25m 48.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (25m) 14s ago 25m 52.1M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (24m) 15s ago 24m 55.0M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (24m) 15s ago 24m 51.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (24m) 15s ago 24m 52.7M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (23m) 15s ago 23m 55.8M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (37s) 15s ago 22m 53.7M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (21m) 14s ago 21m 76.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:55:41.664 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (21m) 15s ago 21m 78.5M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:55:42.056 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:55:42.057 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:55:42.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:42.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:41 smithi172 ceph-mon[24813]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:42.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:42.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:41 smithi172 ceph-mon[29218]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:41 smithi177 ceph-mon[25767]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr" 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading grafana daemons", 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:55:42.385 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:55:42.829 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:42 smithi177 ceph-mon[25767]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:42.829 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:42 smithi177 ceph-mon[25767]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:42.829 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:42 smithi177 ceph-mon[25767]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:42.829 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:42 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3684113215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[29218]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[29218]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[29218]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3684113215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:43.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[24813]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:55:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[24813]: from='client.25414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[24813]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:42 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3684113215' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:44.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:44 smithi177 ceph-mon[25767]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:44 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:44 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:44 smithi172 ceph-mon[29218]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:44 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:44 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:44 smithi172 ceph-mon[24813]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:55:44.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:44 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:44.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:44 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:45 smithi177 ceph-mon[25767]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:45 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:45 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-mon[24813]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:45.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-mon[29218]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:45.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:45.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:55:45] "GET /metrics HTTP/1.1" 200 37998 "" "Prometheus/2.43.0" 2024-03-21T12:55:47.418 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:47 smithi177 ceph-mon[25767]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:47.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:47 smithi172 ceph-mon[24813]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:47.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:47 smithi172 ceph-mon[29218]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:55:48.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:48.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:48.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:48.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:48.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:48.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:48 smithi177 ceph-mon[25767]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:55:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:48.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[29218]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:48.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:48 smithi172 ceph-mon[24813]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:55:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:55:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:55:49.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:55:49.650 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:55:49.400+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:49.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:49.653 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:49.654 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:49.654 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:49.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:55:49.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:55:49.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:55:49.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:49.656 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:49.657 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:49.658 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:49.659 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.736 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.737 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:55:49.738 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.739 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:55:49.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:55:49.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:55:49.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:49.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:49.740 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:49 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:50.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:55:50.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:50.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: Upgrade: Finalizing container_image settings 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: Upgrade: Complete! 2024-03-21T12:55:50.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:50.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:50 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.897 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:50.898 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:50.899 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:50.900 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:50.901 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:50.902 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:50.903 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: Upgrade: Finalizing container_image settings 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: Upgrade: Complete! 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:50.904 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:50.905 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: Upgrade: Finalizing container_image settings 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: Upgrade: Complete! 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:55:50.906 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:50 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:51.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:55:51.245+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:51.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:51.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:55:51.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:51.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:51.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:51 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:51.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:51.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:51.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:51.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:55:51.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:51 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:55:53.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.009 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:53.009 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.009 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.009 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.010 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:53.011 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.012 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[24813]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-21T12:55:53.013 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[24813]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.014 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.015 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:53.016 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.017 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[29218]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[29218]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:53.018 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:52 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:52 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:55:53.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:52 smithi177 ceph-mon[25767]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-21T12:55:53.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:52 smithi177 ceph-mon[25767]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:55:53.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:52 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:55:53.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:52 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:55:53.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:55:53.125+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:53.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:55:53.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:53.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:53.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:53.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:53.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:53.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:53.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:53 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:54.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:54.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:54.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:54.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:54.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:53 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:54.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:55:54.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:55:54.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:55:54.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:55:54.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:55:55.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:54 smithi177 ceph-mon[25767]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-03-21T12:55:55.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:54 smithi172 ceph-mon[24813]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-03-21T12:55:55.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:54 smithi172 ceph-mon[29218]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-03-21T12:55:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:55:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:55:55] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.43.0" 2024-03-21T12:55:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:56 smithi177 ceph-mon[25767]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-21T12:55:57.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:56 smithi172 ceph-mon[24813]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-21T12:55:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:56 smithi172 ceph-mon[29218]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-21T12:55:59.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:55:58 smithi177 ceph-mon[25767]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:55:59.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:55:58 smithi172 ceph-mon[24813]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:55:59.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:55:58 smithi172 ceph-mon[29218]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:01.064 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:00 smithi172 ceph-mon[24813]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:56:01.065 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:00 smithi172 ceph-mon[29218]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:56:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:00 smithi177 ceph-mon[25767]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:56:02.061 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:01 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:02 smithi177 ceph-mon[25767]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:56:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:02 smithi172 ceph-mon[24813]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:56:03.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:02 smithi172 ceph-mon[29218]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:56:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:04 smithi177 ceph-mon[25767]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:04 smithi172 ceph-mon[24813]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:05.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:04 smithi172 ceph-mon[29218]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:56:05] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.43.0" 2024-03-21T12:56:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:05 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:05 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:05 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:07.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:06 smithi177 ceph-mon[25767]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:07.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:06 smithi172 ceph-mon[29218]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:06 smithi172 ceph-mon[24813]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:08 smithi177 ceph-mon[25767]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:08 smithi172 ceph-mon[24813]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:08 smithi172 ceph-mon[29218]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:10 smithi177 ceph-mon[25767]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:10 smithi172 ceph-mon[24813]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:10 smithi172 ceph-mon[29218]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:11.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:11 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:13.028 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:12 smithi172 ceph-mon[24813]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:13.028 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:12 smithi172 ceph-mon[29218]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:13.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:12 smithi177 ceph-mon[25767]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:13.358 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-21T12:56:13.940 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:13 smithi172 ceph-mon[24813]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:13.940 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:13 smithi172 ceph-mon[29218]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:13 smithi177 ceph-mon[25767]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:14.918 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:14 smithi172 ceph-mon[24813]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:14.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:14 smithi172 ceph-mon[29218]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:15.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:14 smithi177 ceph-mon[25767]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:15.425 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:56:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:56:15] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.43.0" 2024-03-21T12:56:16.025 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:15 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1431404065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:16.026 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:15 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1431404065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:16.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-21T12:56:16.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:15 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1431404065' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:16 smithi177 ceph-mon[25767]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:16 smithi172 ceph-mon[24813]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:17.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:16 smithi172 ceph-mon[29218]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:18.044 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)" 2024-03-21T12:56:18.570 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-21T12:56:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:18 smithi177 ceph-mon[25767]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:18 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1883427913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:19.134 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:18 smithi172 ceph-mon[24813]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:19.134 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:18 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1883427913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:19.134 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:18 smithi172 ceph-mon[29218]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:19.134 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:18 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1883427913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:20.671 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:56:20.960 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:20 smithi172 ceph-mon[24813]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:20.960 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:20 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:20.960 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:20 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/670449210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:20.960 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:20 smithi172 ceph-mon[29218]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:20.960 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:20 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:20.961 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:20 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/670449210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:20 smithi177 ceph-mon[25767]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:20 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:20 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/670449210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:21.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-21T12:56:21.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:21.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:22.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:21 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:22.957 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:22 smithi172 ceph-mon[24813]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:22.957 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:22 smithi172 ceph-mon[29218]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:22 smithi177 ceph-mon[25767]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:24 smithi172 ceph-mon[29218]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:24 smithi172 ceph-mon[29218]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:24 smithi172 ceph-mon[24813]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:24 smithi172 ceph-mon[24813]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:25.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:24 smithi177 ceph-mon[25767]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:25.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:24 smithi177 ceph-mon[25767]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:25.468 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:56:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:56:25] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.43.0" 2024-03-21T12:56:26.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T12:56:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:26 smithi177 ceph-mon[25767]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:26 smithi172 ceph-mon[24813]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:26 smithi172 ceph-mon[29218]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T12:56:28.053 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T12:56:28.054 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:56:28.054 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:56:28.680 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T12:56:28.839 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:28 smithi172 ceph-mon[24813]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:28.839 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:28 smithi172 ceph-mon[24813]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:28 smithi177 ceph-mon[25767]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:28 smithi177 ceph-mon[25767]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:29.123 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:28 smithi172 ceph-mon[29218]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:29.123 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:28 smithi172 ceph-mon[29218]: from='client.15570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:30.756 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:56:30.757 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:56:30.757 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:56:31.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:30 smithi172 ceph-mon[24813]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:31.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:30 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/3044735394' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:56:31.077 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:30 smithi172 ceph-mon[29218]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:31.077 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:30 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/3044735394' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:56:31.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:30 smithi177 ceph-mon[25767]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:31.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:30 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/3044735394' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:56:31.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-03-21T12:56:31.981 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:31.982 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:32.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:31 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:32 smithi177 ceph-mon[25767]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:33.138 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:32 smithi172 ceph-mon[24813]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:33.138 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:32 smithi172 ceph-mon[29218]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:35.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:34 smithi172 ceph-mon[29218]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:35.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:34 smithi172 ceph-mon[29218]: from='client.25480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:35.005 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:34 smithi172 ceph-mon[29218]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mon", "hosts": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:35.005 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:34 smithi172 ceph-mon[24813]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:35.005 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:34 smithi172 ceph-mon[24813]: from='client.25480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:35.005 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:34 smithi172 ceph-mon[24813]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mon", "hosts": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:35.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:34 smithi177 ceph-mon[25767]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:35.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:34 smithi177 ceph-mon[25767]: from='client.25480 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:34 smithi177 ceph-mon[25767]: from='client.15588 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mon", "hosts": "smithi177", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:56:35] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.43.0" 2024-03-21T12:56:35.828 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:35 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:36.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T12:56:37.130 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:37.131 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:37.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:37.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:36 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:36 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:38.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[24813]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:38.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[24813]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:38.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:38.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:38.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[29218]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:38.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[29218]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:38.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:38.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:37 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:37 smithi177 ceph-mon[25767]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:37 smithi177 ceph-mon[25767]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:37 smithi177 ceph-mon[25767]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:37 smithi177 ceph-mon[25767]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:39.588 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:56:39.589 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:56:39.390+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:39.589 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:56:39.589 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:39.590 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:39.591 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:39.592 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:39.593 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:39.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:39.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:39.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:39.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:39.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:39.594 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-mon[24813]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:39 smithi172 ceph-mon[29218]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:40.257 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:56:40.257 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (78s) 73s ago 23m 16.7M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:56:40.257 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (62s) 57s ago 22m 51.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:56:40.257 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 73s ago 22m 64.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (2m) 57s ago 27m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (12m) 73s ago 29m 504M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (29m) 73s ago 29m 80.8M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (27m) 57s ago 27m 55.5M 2048M 17.2.0 e1d6a67b021e a6bfda0a2e2b 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (28m) 73s ago 28m 63.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (119s) 73s ago 23m 8354k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (113s) 57s ago 23m 14.7M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (27m) 73s ago 27m 56.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (26m) 73s ago 26m 57.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (26m) 73s ago 26m 48.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (26m) 73s ago 26m 52.1M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (25m) 57s ago 25m 55.0M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (25m) 57s ago 25m 51.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (25m) 57s ago 25m 52.7M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:56:40.258 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (24m) 57s ago 24m 55.8M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:56:40.259 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (95s) 57s ago 23m 56.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:56:40.259 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (22m) 73s ago 22m 76.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:56:40.259 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (22m) 57s ago 22m 78.5M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:56:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:39 smithi177 ceph-mon[25767]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:40.651 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:56:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:56:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-21T12:56:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:56:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:56:40.651 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:56:40.652 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:56:40.653 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:40.974 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:40.975 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:40.976 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:40.977 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:40.978 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.979 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:40.980 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:40.981 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:40.982 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:40.983 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:40.983 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:40.983 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:40.983 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.983 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.984 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[24813]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:40.984 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[24813]: from='client.25492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:40.984 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[24813]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:40.984 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/1206649669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:40.984 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[29218]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[29218]: from='client.25492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[29218]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:40.985 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:40 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/1206649669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi177", 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "", 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 image", 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:56:40.987 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:56:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:40 smithi177 ceph-mon[25767]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:41.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:41.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:41.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:40 smithi177 ceph-mon[25767]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:40 smithi177 ceph-mon[25767]: from='client.25492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:40 smithi177 ceph-mon[25767]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:41.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:40 smithi177 ceph-mon[25767]: from='client.? 172.21.15.172:0/1206649669' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:42.046 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[24813]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:42.046 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:42.046 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[24813]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:42.046 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[24813]: from='client.25516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[29218]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[29218]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:42.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:41 smithi172 ceph-mon[29218]: from='client.25516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:41 smithi177 ceph-mon[25767]: from='client.25504 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:41 smithi177 ceph-mon[25767]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:41 smithi177 ceph-mon[25767]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:42.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:41 smithi177 ceph-mon[25767]: from='client.25516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:56:43.328 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:43.328 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: Upgrade: Setting container_image for all mgr 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-21T12:56:43.329 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:43 smithi177 ceph-mon[25767]: Upgrade: It appears safe to stop mon.b 2024-03-21T12:56:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:56:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:56:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all mgr 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[24813]: Upgrade: It appears safe to stop mon.b 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all mgr 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:56:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-21T12:56:43.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:43 smithi172 ceph-mon[29218]: Upgrade: It appears safe to stop mon.b 2024-03-21T12:56:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[24813]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[29218]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:56:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:44 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:44.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:44 smithi177 ceph-mon[25767]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:44.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:44 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:44.439 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:44 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:56:44.440 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:44 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:56:44.440 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:44 smithi177 ceph-mon[25767]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:45.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:45 smithi177 ceph-mon[25767]: Upgrade: Updating mon.b 2024-03-21T12:56:45.300 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:45 smithi177 ceph-mon[25767]: Deploying daemon mon.b on smithi177 2024-03-21T12:56:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:45 smithi172 ceph-mon[24813]: Upgrade: Updating mon.b 2024-03-21T12:56:45.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:45 smithi172 ceph-mon[24813]: Deploying daemon mon.b on smithi177 2024-03-21T12:56:45.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:56:45] "GET /metrics HTTP/1.1" 200 37998 "" "Prometheus/2.43.0" 2024-03-21T12:56:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:45 smithi172 ceph-mon[29218]: Upgrade: Updating mon.b 2024-03-21T12:56:45.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:45 smithi172 ceph-mon[29218]: Deploying daemon mon.b on smithi177 2024-03-21T12:56:45.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:45 smithi177 systemd[1]: Stopping Ceph mon.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:56:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b[25763]: 2024-03-21T12:56:45.729+0000 7fd4d6434700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T12:56:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b[25763]: 2024-03-21T12:56:45.729+0000 7fd4d6434700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-21T12:56:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:45 smithi177 podman[67802]: 2024-03-21 12:56:45.8730887 +0000 UTC m=+0.246499481 container died a6bfda0a2e2bd6d74db661b32aaaa7e20356c4dda845a469ecbb68ddce024d1e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=-17.2.0, version=8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., ceph=True, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870) 2024-03-21T12:56:46.472 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 podman[67802]: 2024-03-21 12:56:46.221203052 +0000 UTC m=+0.594613829 container cleanup a6bfda0a2e2bd6d74db661b32aaaa7e20356c4dda845a469ecbb68ddce024d1e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, vcs-type=git, architecture=x86_64, release=754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, name=centos-stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8) 2024-03-21T12:56:46.472 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 bash[67802]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b 2024-03-21T12:56:46.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 podman[67816]: 2024-03-21 12:56:46.47244088 +0000 UTC m=+0.599633208 container remove a6bfda0a2e2bd6d74db661b32aaaa7e20356c4dda845a469ecbb68ddce024d1e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, vcs-type=git, release=754, RELEASE=HEAD, GIT_BRANCH=HEAD, architecture=x86_64, ceph=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-21T12:56:46.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-21T12:56:46.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 systemd[1]: Stopped Ceph mon.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:56:46.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.b.service: Consumed 17.051s CPU time. 2024-03-21T12:56:47.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:46 smithi177 systemd[1]: Starting Ceph mon.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:56:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 podman[67947]: 2024-03-21 12:56:46.914140004 +0000 UTC m=+0.018498263 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:56:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 podman[67947]: 2024-03-21 12:56:47.014929072 +0000 UTC m=+0.119287331 container create bb78c25a637538feab6e0ee583e5c91fcac41b2f366c3621e1f8ed6f6c4a1413 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 podman[67947]: 2024-03-21 12:56:47.25938005 +0000 UTC m=+0.363738326 container init bb78c25a637538feab6e0ee583e5c91fcac41b2f366c3621e1f8ed6f6c4a1413 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 podman[67947]: 2024-03-21 12:56:47.262554845 +0000 UTC m=+0.366913107 container start bb78c25a637538feab6e0ee583e5c91fcac41b2f366c3621e1f8ed6f6c4a1413 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 2 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: pidfile_write: ignore empty --pid-file 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: load: jerasure load: lrc 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: RocksDB version: 7.9.2 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Git sha 0 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: DB SUMMARY 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: DB Session ID: T07KL36EOZ4QKZX584SN 2024-03-21T12:56:47.535 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2828 Bytes 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2626346 ; 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.env: 0x55bb9b12cb20 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.info_log: 0x55bb9bd5d920 2024-03-21T12:56:47.536 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.statistics: (nil) 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.use_fsync: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:56:47.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.db_log_dir: 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.wal_dir: 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.write_buffer_manager: 0x55bb9bd2e820 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:56:47.539 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.unordered_write: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.row_cache: None 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.wal_filter: None 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:56:47.540 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.wal_compression: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:56:47.541 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_open_files: -1 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:56:47.542 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Compression algorithms supported: 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kZSTD supported: 0 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:56:47.543 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.merge_operator: 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_filter: None 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bb9bd5da20) 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: index_type: 0 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: data_block_index_type: 0 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: index_shortening: 1 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: checksum: 4 2024-03-21T12:56:47.544 INFO:journalctl@ceph.mon.b.smithi177.stdout: no_block_cache: 0 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache: 0x55bb9bd61090 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache_options: 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: capacity : 536870912 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: num_shard_bits : 4 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: strict_capacity_limit : 0 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_cache_compressed: (nil) 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: persistent_cache: (nil) 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_size: 4096 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_size_deviation: 10 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_restart_interval: 16 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: index_block_restart_interval: 1 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: metadata_block_size: 4096 2024-03-21T12:56:47.545 INFO:journalctl@ceph.mon.b.smithi177.stdout: partition_filters: 0 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: use_delta_encoding: 1 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: filter_policy: bloomfilter 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: whole_key_filtering: 1 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: verify_compression: 0 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: format_version: 5 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: enable_index_compression: 1 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: block_align: 0 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: max_auto_readahead_size: 262144 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: prepopulate_block_cache: 0 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: initial_auto_readahead_size: 8192 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression: NoCompression 2024-03-21T12:56:47.546 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.num_levels: 7 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:56:47.547 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:56:47.548 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:56:47.549 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:56:47.550 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.ttl: 2592000 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:56:47.551 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28769, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 17eb6d37-5e14-43fb-bde9-3b4e54f6a25a 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025807291900, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025807298856, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1632868, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28774, "largest_seqno": 30278, "table_properties": {"data_size": 1626725, "index_size": 3656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1477, "raw_key_size": 14155, "raw_average_key_size": 24, "raw_value_size": 1614241, "raw_average_value_size": 2754, "num_data_blocks": 164, "num_entries": 586, "num_filter_entries": 586, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711025807, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "17eb6d37-5e14-43fb-bde9-3b4e54f6a25a", "db_session_id": "T07KL36EOZ4QKZX584SN", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025807298957, "job": 1, "event": "recovery_finished"} 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-03-21T12:56:47.552 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bb9be50000 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: DB pointer 0x55bb9be3a000 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** DB Stats ** 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-21T12:56:47.553 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: L0 1/0 1.56 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 506.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: L6 1/0 9.20 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: Sum 2/0 10.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 506.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 506.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** Compaction Stats [default] ** 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 506.2 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T12:56:47.554 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Cumulative compaction: 0.00 GB write, 169.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Interval compaction: 0.00 GB write, 169.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Block cache BinnedLRUCache@0x55bb9bd61090#2 capacity: 512.00 MB usage: 5.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.53 KB,0.000292063%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: starting mon.b rank 2 at public addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] at bind addrs [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???) e3 preinit fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:56:47.555 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).mds e1 new map 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).mds e1 print_map 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout: e1 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout: legacy client fscid: -1 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout: 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout: No filesystems configured 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 bash[67947]: bb78c25a637538feab6e0ee583e5c91fcac41b2f366c3621e1f8ed6f6c4a1413 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 systemd[1]: Started Ceph mon.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:56:47.556 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:47 smithi177 ceph-mon[67966]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:47.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:47 smithi172 ceph-mon[24813]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:47.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:47 smithi172 ceph-mon[29218]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: mon.b calling monitor election 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: mon.c calling monitor election 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: mon.a calling monitor election 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:56:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: fsmap 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: mgrmap e42: y(active, since 2m), standbys: x 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:48.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:48 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:48.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:56:48.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: mon.b calling monitor election 2024-03-21T12:56:48.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: mon.c calling monitor election 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: mon.a calling monitor election 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: fsmap 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: mgrmap e42: y(active, since 2m), standbys: x 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:48.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: mon.b calling monitor election 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: mon.c calling monitor election 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: mon.a calling monitor election 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: fsmap 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: mgrmap e42: y(active, since 2m), standbys: x 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:48.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:48.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:48 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:49.831 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:49 smithi177 ceph-mon[67966]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:49.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:49 smithi172 ceph-mon[24813]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:49.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:49 smithi172 ceph-mon[29218]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:50.826 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:50 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:50.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:50 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:50.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:50 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:56:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:51 smithi177 ceph-mon[67966]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:51.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:51 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:51.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:51 smithi172 ceph-mon[24813]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:51.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:56:51.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:51 smithi172 ceph-mon[29218]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:52.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:56:52.305+0000 7f9fb974e700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-21T12:56:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:52 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:52 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:52 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:52 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:52 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:52 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:54.148 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:53 smithi177 ceph-mon[67966]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:54.148 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:53 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:54.148 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:53 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:53 smithi172 ceph-mon[24813]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:53 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:53 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:54.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:53 smithi172 ceph-mon[29218]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:56:54.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:53 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:54.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:53 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:56:55] "GET /metrics HTTP/1.1" 200 37948 "" "Prometheus/2.43.0" 2024-03-21T12:56:56.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:55 smithi177 ceph-mon[67966]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:55 smithi172 ceph-mon[24813]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:55 smithi172 ceph-mon[29218]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: Detected new or changed devices on smithi177 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:57.610 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:57.611 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:57 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi177 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-21T12:56:57.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: Detected new or changed devices on smithi177 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 885 B/s rd, 0 op/s 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 535 B/s rd, 0 op/s 2024-03-21T12:56:57.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:57.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:57.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:57.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:57.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:57 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:56:58.201+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:56:58.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:58.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:58.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:58.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:58.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:56:58 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:58.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:58 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:59.303 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:59.303 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:59.303 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:59.304 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:59.305 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.306 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:59.307 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:59.308 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:56:59.309 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.310 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:59.311 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all crash 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all mds 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all nfs 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all node-exporter 2024-03-21T12:56:59.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all prometheus 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all alertmanager 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all grafana 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all loki 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:59.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all crash 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all mds 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all nfs 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all node-exporter 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all prometheus 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all alertmanager 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all grafana 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all loki 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all promtail 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Finalizing container_image settings 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:56:59.316 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.317 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Upgrade: Complete! 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:59.318 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all promtail 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Finalizing container_image settings 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:56:59.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:56:59.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Upgrade: Complete! 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-03-21T12:56:59.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:59.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:59.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:59.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:56:59 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:56:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:56:59.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:56:59.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mds 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:56:59.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nfs 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all node-exporter 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all prometheus 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all alertmanager 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all grafana 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all loki 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all promtail 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Finalizing container_image settings 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:56:59.614 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:56:59.615 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Upgrade: Complete! 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:56:59.616 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 908 B/s rd, 0 op/s 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:56:59.617 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:56:59 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:00.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:57:00.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:00.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:00.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:57:00.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:00.178+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.406 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:00.407 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.408 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.409 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:00 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:00.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:00.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:00.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 930 B/s rd, 0 op/s 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T12:57:00.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:00 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:01.440 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:01.441 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:57:01.441 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:01.441 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:01.441 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:01.441 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:01.441 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:01 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:01.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:01 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:57:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:01 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:01 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:01 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:01.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:01 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:02.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:02.009+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:02.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:02.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:57:02.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:02.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:02.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:02 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:03.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:03.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:03.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:02 smithi177 ceph-mon[67966]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T12:57:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:03.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:03.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:03.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:03.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:03.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:03.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-mon[24813]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T12:57:03.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:03.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:03.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:03.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:03.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:03.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:02 smithi172 ceph-mon[29218]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T12:57:04.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:04 smithi177 ceph-mon[67966]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-03-21T12:57:04.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:04 smithi172 ceph-mon[24813]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-03-21T12:57:04.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:04 smithi172 ceph-mon[29218]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-03-21T12:57:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:05 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:05.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:05 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:05.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:57:05] "GET /metrics HTTP/1.1" 200 37948 "" "Prometheus/2.43.0" 2024-03-21T12:57:05.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:05 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:06.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:06 smithi177 ceph-mon[67966]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-21T12:57:06.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:06 smithi172 ceph-mon[24813]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-21T12:57:06.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:06 smithi172 ceph-mon[29218]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 650 B/s rd, 0 op/s 2024-03-21T12:57:08.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:08 smithi177 ceph-mon[67966]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:57:08.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:08 smithi172 ceph-mon[24813]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:57:08.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:08 smithi172 ceph-mon[29218]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:57:10.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:10 smithi177 ceph-mon[67966]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:57:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:10 smithi172 ceph-mon[24813]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:57:10.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:10 smithi172 ceph-mon[29218]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:57:11.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:11.645 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:11 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:11 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:11.955 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:57:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:12 smithi172 ceph-mon[24813]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:12.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:12 smithi172 ceph-mon[24813]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:12 smithi172 ceph-mon[29218]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:12.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:12 smithi172 ceph-mon[29218]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:13.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:12 smithi177 ceph-mon[67966]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:13.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:12 smithi177 ceph-mon[67966]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:14.078 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:57:14.078 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (112s) 107s ago 23m 16.7M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (95s) 22s ago 23m 74.6M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 107s ago 22m 64.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (2m) 22s ago 28m 450M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (12m) 107s ago 29m 504M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (29m) 107s ago 29m 80.8M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (27s) 22s ago 28m 17.8M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (28m) 107s ago 28m 63.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (2m) 107s ago 24m 8354k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (2m) 22s ago 24m 15.4M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (27m) 107s ago 27m 56.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (27m) 107s ago 27m 57.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (27m) 107s ago 27m 48.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (26m) 107s ago 26m 52.1M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:57:14.079 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (26m) 22s ago 26m 55.0M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:57:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (26m) 22s ago 26m 51.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:57:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (25m) 22s ago 25m 52.7M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:57:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (25m) 22s ago 25m 55.8M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:57:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (2m) 22s ago 24m 63.6M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:57:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (23m) 107s ago 23m 76.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:57:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (22m) 22s ago 22m 78.9M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:57:14.514 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:14 smithi172 ceph-mon[24813]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:14.514 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:14 smithi172 ceph-mon[24813]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:14.515 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:14 smithi172 ceph-mon[29218]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:14.515 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:14 smithi172 ceph-mon[29218]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:14 smithi177 ceph-mon[67966]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:14.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:14 smithi177 ceph-mon[67966]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:14.652 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-03-21T12:57:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:57:15] "GET /metrics HTTP/1.1" 200 38016 "" "Prometheus/2.43.0" 2024-03-21T12:57:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:16 smithi177 ceph-mon[67966]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:16.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:16 smithi172 ceph-mon[24813]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:16.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:16 smithi172 ceph-mon[29218]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:16.709 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:57:17.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:17 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/479809840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:17.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:17 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/479809840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:17.363 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T12:57:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:17 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/479809840' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:18 smithi177 ceph-mon[67966]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:18 smithi172 ceph-mon[24813]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:18.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:18 smithi172 ceph-mon[29218]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:19.375 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:57:19.375 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:57:19.376 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:57:20.055 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T12:57:20.361 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:20 smithi172 ceph-mon[24813]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:20.362 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:20 smithi172 ceph-mon[24813]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:20.362 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:20 smithi172 ceph-mon[29218]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:20.362 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:20 smithi172 ceph-mon[29218]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:20 smithi177 ceph-mon[67966]: from='client.15642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:20.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:20 smithi177 ceph-mon[67966]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:21 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:21 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:21.622 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:21 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:21.622 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:21 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:21.622 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:21 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:21.622 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:21 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:22.117 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:57:22.118 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:57:22.118 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:57:22.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-03-21T12:57:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:22 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2199213345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:57:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:22 smithi177 ceph-mon[67966]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:23.109 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:22 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/2199213345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:57:23.109 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:22 smithi172 ceph-mon[24813]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:23.109 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:22 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/2199213345' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:57:23.109 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:22 smithi172 ceph-mon[29218]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:57:24.377 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:24 smithi172 ceph-mon[24813]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:24.377 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:24 smithi172 ceph-mon[29218]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:24.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:24 smithi177 ceph-mon[67966]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:25.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:25 smithi177 ceph-mon[67966]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:25.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:25 smithi177 ceph-mon[67966]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mon", "hosts": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:25.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:25 smithi172 ceph-mon[24813]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:25.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:25 smithi172 ceph-mon[24813]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mon", "hosts": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:57:25] "GET /metrics HTTP/1.1" 200 38019 "" "Prometheus/2.43.0" 2024-03-21T12:57:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:25 smithi172 ceph-mon[29218]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:25.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:25 smithi172 ceph-mon[29218]: from='client.25558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "mon", "hosts": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:26.374 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:26 smithi172 ceph-mon[24813]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:26.374 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:26 smithi172 ceph-mon[29218]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:26.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:26 smithi177 ceph-mon[67966]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:27.212 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:27.827 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:28.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[24813]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:28.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:28.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:28.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:28.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:28.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:28 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:57:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:28.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:28.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:57:28.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:28.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:28.609 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:28 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:57:30.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-mon[24813]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:30.404 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-mon[29218]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:30.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:30 smithi177 ceph-mon[67966]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:30.758 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:30.594+0000 7f9fb6748700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:30.759 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:30.760 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:30.761 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:30.762 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:30.763 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:30 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.468 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:57:31.468 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (2m) 2m ago 24m 16.7M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (113s) 39s ago 23m 74.6M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 2m ago 23m 64.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (3m) 39s ago 28m 450M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (13m) 2m ago 30m 504M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (30m) 2m ago 30m 80.8M 2048M 17.2.0 e1d6a67b021e b3142cf6ffcb 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (44s) 39s ago 28m 17.8M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (28m) 2m ago 28m 63.4M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (2m) 2m ago 24m 8354k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (2m) 39s ago 24m 15.4M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (28m) 2m ago 28m 56.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (27m) 2m ago 27m 57.6M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (27m) 2m ago 27m 48.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (27m) 2m ago 27m 52.1M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:57:31.469 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (26m) 39s ago 26m 55.0M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:57:31.470 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (26m) 39s ago 26m 51.7M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:57:31.470 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (25m) 39s ago 25m 52.7M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:57:31.470 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (25m) 39s ago 25m 55.8M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:57:31.470 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (2m) 39s ago 24m 63.6M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:57:31.470 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (23m) 2m ago 23m 76.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:57:31.470 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (23m) 39s ago 23m 78.9M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:57:31.559 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:31.559 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:31 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:31.560 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:57:31.561 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:31.562 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:31.563 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:31.564 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.564 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.564 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:31 smithi177 ceph-mon[67966]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:31.564 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:31 smithi177 ceph-mon[67966]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:31.564 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:31 smithi177 ceph-mon[67966]: from='client.25570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[24813]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[24813]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:31.600 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.601 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:57:31.602 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:31.603 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[24813]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[24813]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:31.604 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[24813]: from='client.25570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:31.605 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:57:31.606 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:57:31.607 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:57:31.608 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:57:31.609 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:57:31.609 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:57:31.609 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[29218]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:57:31.609 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[29218]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:31.609 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:31 smithi172 ceph-mon[29218]: from='client.25570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 1 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "mds": {}, 2024-03-21T12:57:31.881 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:57:31.882 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:57:32.227 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi172", 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "", 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 image", 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:57:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:57:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:32 smithi177 ceph-mon[67966]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-21T12:57:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:32 smithi177 ceph-mon[67966]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:32 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/75616275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:32 smithi172 ceph-mon[29218]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-21T12:57:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:32 smithi172 ceph-mon[29218]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:32 smithi172 ceph-mon[29218]: from='client.? 172.21.15.172:0/75616275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:32 smithi172 ceph-mon[24813]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 566 B/s rd, 0 op/s 2024-03-21T12:57:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:32 smithi172 ceph-mon[24813]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:32 smithi172 ceph-mon[24813]: from='client.? 172.21.15.172:0/75616275' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:57:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:57:33.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: Upgrade: Setting container_image for all mgr 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[24813]: Upgrade: It appears safe to stop mon.a 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:57:33.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:33.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:33.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: Upgrade: Setting container_image for all mgr 2024-03-21T12:57:33.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:33.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:57:33.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-21T12:57:33.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:33 smithi172 ceph-mon[29218]: Upgrade: It appears safe to stop mon.a 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mgr 2024-03-21T12:57:34.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:34.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:57:34.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-21T12:57:34.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:33 smithi177 ceph-mon[67966]: Upgrade: It appears safe to stop mon.a 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[24813]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[29218]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:35.100 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:57:35.101 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:57:35.101 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:34 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:35.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:34 smithi177 ceph-mon[67966]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:57:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:34 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' 2024-03-21T12:57:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:34 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:57:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:34 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:57:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:34 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:35.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:57:35] "GET /metrics HTTP/1.1" 200 38019 "" "Prometheus/2.43.0" 2024-03-21T12:57:36.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-mon[24813]: Upgrade: Updating mon.a 2024-03-21T12:57:36.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-mon[24813]: Deploying daemon mon.a on smithi172 2024-03-21T12:57:36.100 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-mon[24813]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:36.100 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-mon[29218]: Upgrade: Updating mon.a 2024-03-21T12:57:36.101 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-mon[29218]: Deploying daemon mon.a on smithi172 2024-03-21T12:57:36.101 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:35 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:35 smithi177 ceph-mon[67966]: Upgrade: Updating mon.a 2024-03-21T12:57:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:35 smithi177 ceph-mon[67966]: Deploying daemon mon.a on smithi172 2024-03-21T12:57:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:35 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:36.351 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:36 smithi172 systemd[1]: Stopping Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:57:36.351 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a[24809]: 2024-03-21T12:57:36.348+0000 7f9c96336700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T12:57:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a[24809]: 2024-03-21T12:57:36.348+0000 7f9c96336700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-21T12:57:36.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:36 smithi172 podman[84458]: 2024-03-21 12:57:36.391393665 +0000 UTC m=+0.148516963 container died b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, io.openshift.tags=base centos centos-stream, architecture=x86_64, RELEASE=HEAD, io.openshift.expose-services=, release=754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-21T12:57:36.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:36] ENGINE Bus STOPPING 2024-03-21T12:57:36.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T12:57:36.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:36] ENGINE Bus STOPPED 2024-03-21T12:57:36.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:36] ENGINE Bus STARTING 2024-03-21T12:57:36.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:36] ENGINE Serving on http://:::9283 2024-03-21T12:57:36.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:36] ENGINE Bus STARTED 2024-03-21T12:57:37.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:36 smithi172 podman[84458]: 2024-03-21 12:57:36.77738724 +0000 UTC m=+0.534510540 container cleanup b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.expose-services=, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, name=centos-stream, architecture=x86_64) 2024-03-21T12:57:37.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:36 smithi172 bash[84458]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a 2024-03-21T12:57:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 podman[84483]: 2024-03-21 12:57:37.059763036 +0000 UTC m=+0.663603146 container remove b3142cf6ffcb96e1ad98482d91830df2f8627305e0cea670900ddfa70955824e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.openshift.tags=base centos centos-stream, ceph=True, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, architecture=x86_64, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-21T12:57:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T12:57:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 systemd[1]: Stopped Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:57:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service: Consumed 25.460s CPU time. 2024-03-21T12:57:37.697 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 systemd[1]: Starting Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:57:37.697 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 podman[84616]: 2024-03-21 12:57:37.611164541 +0000 UTC m=+0.111843942 container create fb5500dccf6a39ba5783434c0cab7f82ab8c1105da245175efc6fb8e6cb7188d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-21T12:57:37.697 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 podman[84616]: 2024-03-21 12:57:37.517683257 +0000 UTC m=+0.018362629 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 podman[84616]: 2024-03-21 12:57:37.847334913 +0000 UTC m=+0.348014272 container init fb5500dccf6a39ba5783434c0cab7f82ab8c1105da245175efc6fb8e6cb7188d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 podman[84616]: 2024-03-21 12:57:37.850457129 +0000 UTC m=+0.351136489 container start fb5500dccf6a39ba5783434c0cab7f82ab8c1105da245175efc6fb8e6cb7188d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 2 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: pidfile_write: ignore empty --pid-file 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: load: jerasure load: lrc 2024-03-21T12:57:38.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: RocksDB version: 7.9.2 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Git sha 0 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: DB SUMMARY 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: DB Session ID: MOWQNNHSGB9XH2JV7QKZ 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3094 Bytes 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 177577 ; 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:57:38.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T12:57:38.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.env: 0x56269d4acb20 2024-03-21T12:57:38.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T12:57:38.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.info_log: 0x56269dfbd920 2024-03-21T12:57:38.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:57:38.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.statistics: (nil) 2024-03-21T12:57:38.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.use_fsync: 0 2024-03-21T12:57:38.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:57:38.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:57:38.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:57:38.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.db_log_dir: 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.wal_dir: 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:57:38.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.write_buffer_manager: 0x56269df8e820 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.unordered_write: 0 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.row_cache: None 2024-03-21T12:57:38.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.wal_filter: None 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.wal_compression: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:57:38.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_open_files: -1 2024-03-21T12:57:38.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Compression algorithms supported: 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kZSTD supported: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:57:38.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.merge_operator: 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_filter: None 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56269dfbda20) 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:57:38.156 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_type: 0 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: data_block_index_type: 0 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_shortening: 1 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: checksum: 4 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: no_block_cache: 0 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache: 0x56269dfc1090 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_options: 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: capacity : 536870912 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: num_shard_bits : 4 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: strict_capacity_limit : 0 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_compressed: (nil) 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: persistent_cache: (nil) 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_size: 4096 2024-03-21T12:57:38.157 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_size_deviation: 10 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_restart_interval: 16 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_block_restart_interval: 1 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: metadata_block_size: 4096 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: partition_filters: 0 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: use_delta_encoding: 1 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: filter_policy: bloomfilter 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: whole_key_filtering: 1 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: verify_compression: 0 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: format_version: 5 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: enable_index_compression: 1 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_align: 0 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: max_auto_readahead_size: 262144 2024-03-21T12:57:38.158 INFO:journalctl@ceph.mon.a.smithi172.stdout: prepopulate_block_cache: 0 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout: initial_auto_readahead_size: 8192 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression: NoCompression 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.num_levels: 7 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:57:38.159 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:57:38.160 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:57:38.161 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:57:38.162 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.ttl: 2592000 2024-03-21T12:57:38.163 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29243, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T12:57:38.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4a072e34-c1ac-47d3-bd13-f5d5fc2d2cc7 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025857879644, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025857880904, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 175997, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29245, "largest_seqno": 29400, "table_properties": {"data_size": 174268, "index_size": 461, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2411, "raw_average_key_size": 25, "raw_value_size": 172230, "raw_average_value_size": 1832, "num_data_blocks": 21, "num_entries": 94, "num_filter_entries": 94, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711025857, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4a072e34-c1ac-47d3-bd13-f5d5fc2d2cc7", "db_session_id": "MOWQNNHSGB9XH2JV7QKZ", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025857880986, "job": 1, "event": "recovery_finished"} 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56269e0b0000 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: DB pointer 0x56269e09a000 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** DB Stats ** 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:57:38.165 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: L0 1/0 171.87 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 188.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: L6 1/0 9.99 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Sum 2/0 10.15 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 188.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 188.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:57:38.166 INFO:journalctl@ceph.mon.a.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 188.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative compaction: 0.00 GB write, 50.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval compaction: 0.00 GB write, 50.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Block cache BinnedLRUCache@0x56269dfc1090#2 capacity: 512.00 MB usage: 0.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.55 KB,0.000104308%) Misc(1,0.00 KB,0%) 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:57:38.167 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: starting mon.a rank 0 at public addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] at bind addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???) e3 preinit fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).mds e1 new map 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).mds e1 print_map 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout: e1 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout: legacy client fscid: -1 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout: No filesystems configured 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 ceph-mon[84635]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 bash[84616]: fb5500dccf6a39ba5783434c0cab7f82ab8c1105da245175efc6fb8e6cb7188d 2024-03-21T12:57:38.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:37 smithi172 systemd[1]: Started Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:57:41.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ignoring --setuser ceph since I am not root 2024-03-21T12:57:41.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ignoring --setgroup ceph since I am not root 2024-03-21T12:57:41.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mgr[63157]: -- 172.21.15.172:0/4039016708 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5602196845a0 con 0x560219658c00 2024-03-21T12:57:41.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: mon.a calling monitor election 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: fsmap 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:57:41.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: from='mgr.25240 ' entity='' 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[29218]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: mon.a calling monitor election 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:57:41.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: fsmap 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: from='mgr.25240 ' entity='' 2024-03-21T12:57:41.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:40 smithi172 ceph-mon[84635]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: ignoring --setuser ceph since I am not root 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: ignoring --setgroup ceph since I am not root 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mgr[61380]: -- 172.21.15.177:0/3553104980 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557307dde5a0 con 0x557307db3000 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: from='mgr.25240 172.21.15.172:0/574818336' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:57:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: mon.a calling monitor election 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: fsmap 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: from='mgr.25240 ' entity='' 2024-03-21T12:57:41.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:40 smithi177 ceph-mon[67966]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-21T12:57:41.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:41.506+0000 7fe1fe7c0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:57:41.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:41.602+0000 7fe1fe7c0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:57:41.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:41.516+0000 7f0d50a6a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T12:57:41.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:41 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:41.612+0000 7f0d50a6a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T12:57:42.576 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:42.245+0000 7fe1fe7c0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:57:42.576 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:42.338+0000 7fe1fe7c0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:57:42.577 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:42.480+0000 7fe1fe7c0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:57:42.577 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:42.576+0000 7fe1fe7c0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:57:42.595 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:42.260+0000 7f0d50a6a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T12:57:42.595 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:42.354+0000 7f0d50a6a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T12:57:42.595 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:42.499+0000 7f0d50a6a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T12:57:42.596 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:42.595+0000 7f0d50a6a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T12:57:42.858 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:42.661+0000 7fe1fe7c0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:57:42.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:42.680+0000 7f0d50a6a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T12:57:43.219 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:42.930+0000 7fe1fe7c0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:57:43.220 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:43.039+0000 7fe1fe7c0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:57:43.220 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:43.125+0000 7fe1fe7c0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:57:43.220 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:42 smithi177 ceph-mon[67966]: mgrmap e44: y(active, since 3m), standbys: x 2024-03-21T12:57:43.249 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-mon[29218]: mgrmap e44: y(active, since 3m), standbys: x 2024-03-21T12:57:43.249 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-mon[84635]: mgrmap e44: y(active, since 3m), standbys: x 2024-03-21T12:57:43.249 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:42 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:42.952+0000 7f0d50a6a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T12:57:43.250 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:43.063+0000 7f0d50a6a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T12:57:43.250 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:43.151+0000 7f0d50a6a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T12:57:43.250 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:43.248+0000 7f0d50a6a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:57:43.607 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:43.219+0000 7fe1fe7c0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T12:57:43.608 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:43.397+0000 7fe1fe7c0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:57:43.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:43.428+0000 7f0d50a6a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T12:57:44.832 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:44.359+0000 7fe1fe7c0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:57:44.833 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:44.491+0000 7fe1fe7c0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:57:44.880 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:44.404+0000 7f0d50a6a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T12:57:44.880 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:44.539+0000 7f0d50a6a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T12:57:44.881 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:44.880+0000 7f0d50a6a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:57:45.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:44.830+0000 7fe1fe7c0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T12:57:45.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:44.920+0000 7fe1fe7c0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:57:45.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:44.974+0000 7f0d50a6a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T12:57:45.601 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:45.119+0000 7fe1fe7c0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:57:45.601 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:45.310+0000 7fe1fe7c0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:57:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:45.177+0000 7f0d50a6a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:57:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:45.370+0000 7f0d50a6a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T12:57:45.857 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:45.600+0000 7fe1fe7c0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:57:45.933 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:45.661+0000 7f0d50a6a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T12:57:46.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:45.865+0000 7fe1fe7c0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:57:46.358 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:46.107+0000 7fe1fe7c0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:57:46.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:45.929+0000 7f0d50a6a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T12:57:46.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:46.173+0000 7f0d50a6a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T12:57:47.357 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:46.918+0000 7fe1fe7c0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:57:47.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:46.999+0000 7f0d50a6a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T12:57:47.813 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:47.491+0000 7fe1fe7c0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:57:47.813 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:47.591+0000 7fe1fe7c0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:57:47.891 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:47.584+0000 7f0d50a6a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T12:57:47.891 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:47.684+0000 7f0d50a6a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-mon[67966]: Standby manager daemon x restarted 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-mon[67966]: Standby manager daemon x started 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-mon[67966]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-mon[67966]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-mon[67966]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-mon[67966]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:57:48.108 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:47.812+0000 7fe1fe7c0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:57:48.109 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: [21/Mar/2024:12:57:47] ENGINE Bus STARTING 2024-03-21T12:57:48.109 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: CherryPy Checker: 2024-03-21T12:57:48.109 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: The Application mounted at '' has an empty config. 2024-03-21T12:57:48.109 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: 2024-03-21T12:57:48.109 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: [21/Mar/2024:12:57:47] ENGINE Serving on http://:::9283 2024-03-21T12:57:48.109 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 12:57:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x[61376]: [21/Mar/2024:12:57:47] ENGINE Bus STARTED 2024-03-21T12:57:48.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:47.915+0000 7f0d50a6a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T12:57:48.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:48] ENGINE Bus STARTING 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[29218]: Standby manager daemon x restarted 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[29218]: Standby manager daemon x started 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[29218]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[84635]: Standby manager daemon x restarted 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[84635]: Standby manager daemon x started 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[84635]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[84635]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[84635]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T12:57:48.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:47 smithi172 ceph-mon[84635]: from='mgr.? 172.21.15.177:0/219791521' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T12:57:48.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CherryPy Checker: 2024-03-21T12:57:48.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: The Application mounted at '' has an empty config. 2024-03-21T12:57:48.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:48.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:48] ENGINE Serving on http://:::9283 2024-03-21T12:57:48.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:12:57:48] ENGINE Bus STARTED 2024-03-21T12:57:49.143 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:48.941+0000 7f0d06c0d700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-21T12:57:49.143 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: mgrmap e45: y(active, since 3m), standbys: x 2024-03-21T12:57:49.143 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: Active manager daemon y restarted 2024-03-21T12:57:49.143 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: Activating manager daemon y 2024-03-21T12:57:49.143 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:57:49.143 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: mgrmap e46: y(active, starting, since 0.00718248s), standbys: x 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:57:49.144 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: Manager daemon y is now available 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: mgrmap e45: y(active, since 3m), standbys: x 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: Active manager daemon y restarted 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: Activating manager daemon y 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: mgrmap e46: y(active, starting, since 0.00718248s), standbys: x 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:57:49.145 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:57:49.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:57:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: Manager daemon y is now available 2024-03-21T12:57:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:57:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:48 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: mgrmap e45: y(active, since 3m), standbys: x 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: Active manager daemon y restarted 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: Activating manager daemon y 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: mgrmap e46: y(active, starting, since 0.00718248s), standbys: x 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:57:49.164 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T12:57:49.165 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: Manager daemon y is now available 2024-03-21T12:57:49.166 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:57:49.166 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:57:49.166 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T12:57:49.166 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T12:57:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[84635]: [21/Mar/2024:12:57:48] ENGINE Bus STARTING 2024-03-21T12:57:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[84635]: [21/Mar/2024:12:57:48] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:57:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[84635]: [21/Mar/2024:12:57:48] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:57:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[84635]: [21/Mar/2024:12:57:48] ENGINE Bus STARTED 2024-03-21T12:57:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[84635]: mgrmap e47: y(active, since 1.0095s), standbys: x 2024-03-21T12:57:50.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[84635]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:50.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[29218]: [21/Mar/2024:12:57:48] ENGINE Bus STARTING 2024-03-21T12:57:50.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[29218]: [21/Mar/2024:12:57:48] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:57:50.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[29218]: [21/Mar/2024:12:57:48] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:57:50.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[29218]: [21/Mar/2024:12:57:48] ENGINE Bus STARTED 2024-03-21T12:57:50.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[29218]: mgrmap e47: y(active, since 1.0095s), standbys: x 2024-03-21T12:57:50.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:49 smithi172 ceph-mon[29218]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:50.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:49 smithi177 ceph-mon[67966]: [21/Mar/2024:12:57:48] ENGINE Bus STARTING 2024-03-21T12:57:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:49 smithi177 ceph-mon[67966]: [21/Mar/2024:12:57:48] ENGINE Serving on https://172.21.15.172:7150 2024-03-21T12:57:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:49 smithi177 ceph-mon[67966]: [21/Mar/2024:12:57:48] ENGINE Serving on http://172.21.15.172:8765 2024-03-21T12:57:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:49 smithi177 ceph-mon[67966]: [21/Mar/2024:12:57:48] ENGINE Bus STARTED 2024-03-21T12:57:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:49 smithi177 ceph-mon[67966]: mgrmap e47: y(active, since 1.0095s), standbys: x 2024-03-21T12:57:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:49 smithi177 ceph-mon[67966]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:51.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:50 smithi177 ceph-mon[67966]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:57:51.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:50 smithi177 ceph-mon[67966]: Cluster is now healthy 2024-03-21T12:57:51.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:50 smithi177 ceph-mon[67966]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:51.260 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:50 smithi172 ceph-mon[84635]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:57:51.260 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:50 smithi172 ceph-mon[84635]: Cluster is now healthy 2024-03-21T12:57:51.260 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:50 smithi172 ceph-mon[84635]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:51.260 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:50 smithi172 ceph-mon[29218]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T12:57:51.260 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:50 smithi172 ceph-mon[29218]: Cluster is now healthy 2024-03-21T12:57:51.260 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:50 smithi172 ceph-mon[29218]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:52.297 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:52.297 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:51 smithi177 ceph-mon[67966]: mgrmap e48: y(active, since 3s), standbys: x 2024-03-21T12:57:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:51 smithi172 ceph-mon[84635]: mgrmap e48: y(active, since 3s), standbys: x 2024-03-21T12:57:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:51 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:57:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:51 smithi172 ceph-mon[29218]: mgrmap e48: y(active, since 3s), standbys: x 2024-03-21T12:57:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:52 smithi177 ceph-mon[67966]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[84635]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[29218]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:53.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:52 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[84635]: mgrmap e49: y(active, since 5s), standbys: x 2024-03-21T12:57:54.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.242 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[29218]: mgrmap e49: y(active, since 5s), standbys: x 2024-03-21T12:57:54.242 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.242 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.242 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.242 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:53 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.298 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:53 smithi177 ceph-mon[67966]: mgrmap e49: y(active, since 5s), standbys: x 2024-03-21T12:57:54.298 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.298 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.298 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:54.298 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:55.247 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:54 smithi172 ceph-mon[84635]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:55.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:54 smithi172 ceph-mon[29218]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:55.273 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:54 smithi177 ceph-mon[67966]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-21T12:57:55.558 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:57:55] "GET /metrics HTTP/1.1" 200 35217 "" "Prometheus/2.43.0" 2024-03-21T12:57:57.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:56 smithi172 ceph-mon[84635]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:57:57.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:57.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:56 smithi172 ceph-mon[29218]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:57:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:56 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:57.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:56 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:57.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:56 smithi177 ceph-mon[67966]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T12:57:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:57 smithi177 ceph-mon[67966]: Detected new or changed devices on smithi172 2024-03-21T12:57:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[84635]: Detected new or changed devices on smithi172 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[29218]: Detected new or changed devices on smithi172 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:57 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:57:59.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:57:59.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:57:59.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:57:59.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:57:59.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:57:59.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:57:59.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:59.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:58 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:57:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-21T12:57:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi177:/etc/ceph/ceph.conf 2024-03-21T12:57:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi177:/etc/ceph/ceph.client.admin.keyring 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: Updating smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:57:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:57:59.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:58:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:00.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:00.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:57:59.911+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:00.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:00.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:58:00.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:00.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:00.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:57:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:00.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-21T12:58:00.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:00.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:00.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T12:58:00.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:00.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:00.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-21T12:58:00.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:57:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:00 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.112 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.113 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:01.114 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:01.115 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.116 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:00 smithi172 ceph-mon[84635]: Upgrade: It appears safe to stop mon.c 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:00 smithi172 ceph-mon[29218]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:01.117 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:01.118 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:01.119 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:01.120 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.121 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.121 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:00 smithi172 ceph-mon[29218]: Upgrade: It appears safe to stop mon.c 2024-03-21T12:58:01.270 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:00 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:01.270 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:01.270 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:01.271 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:01.272 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:01.273 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:01.274 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:00 smithi177 ceph-mon[67966]: Upgrade: It appears safe to stop mon.c 2024-03-21T12:58:02.290 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[84635]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[29218]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[29218]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:02.291 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:01 smithi172 ceph-mon[29218]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:01 smithi177 ceph-mon[67966]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:58:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:02.581 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:58:03.264 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:02 smithi172 ceph-mon[84635]: Upgrade: Updating mon.c 2024-03-21T12:58:03.264 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:02 smithi172 ceph-mon[84635]: Deploying daemon mon.c on smithi172 2024-03-21T12:58:03.264 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:02 smithi172 ceph-mon[29218]: Upgrade: Updating mon.c 2024-03-21T12:58:03.264 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:02 smithi172 ceph-mon[29218]: Deploying daemon mon.c on smithi172 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (2m) 10s ago 24m 17.7M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (2m) 10s ago 24m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 10s ago 23m 67.7M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (3m) 10s ago 29m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (13m) 10s ago 30m 474M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (25s) 10s ago 30m 30.8M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (76s) 10s ago 29m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T12:58:03.287 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (29m) 10s ago 29m 65.2M 2048M 17.2.0 e1d6a67b021e 6a632c220c5b 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (3m) 10s ago 25m 9101k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (3m) 10s ago 25m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (28m) 10s ago 28m 56.9M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (28m) 10s ago 28m 57.9M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (27m) 10s ago 27m 49.3M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (27m) 10s ago 27m 52.3M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (27m) 10s ago 27m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (26m) 10s ago 26m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (26m) 10s ago 26m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (26m) 10s ago 26m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (2m) 10s ago 25m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (23m) 10s ago 23m 76.7M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:58:03.288 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (23m) 10s ago 23m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:58:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:02 smithi177 ceph-mon[67966]: Upgrade: Updating mon.c 2024-03-21T12:58:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:02 smithi177 ceph-mon[67966]: Deploying daemon mon.c on smithi172 2024-03-21T12:58:03.516 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:03 smithi172 systemd[1]: Stopping Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:58:03.706 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:58:03.707 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-03-21T12:58:03.708 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 4 2024-03-21T12:58:03.708 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:58:03.708 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:58:03.883 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c[29214]: 2024-03-21T12:58:03.515+0000 7f6d7ffed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T12:58:03.883 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c[29214]: 2024-03-21T12:58:03.515+0000 7f6d7ffed700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-21T12:58:03.883 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:03 smithi172 podman[89915]: 2024-03-21 12:58:03.627970269 +0000 UTC m=+0.206558495 container died 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi172", 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "1/2 daemons upgraded", 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading mon daemons", 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:58:04.049 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:58:04.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:03 smithi172 podman[89915]: 2024-03-21 12:58:03.982548717 +0000 UTC m=+0.561136943 container cleanup 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, ceph=True, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8) 2024-03-21T12:58:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:03 smithi172 bash[89915]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c 2024-03-21T12:58:04.616 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 podman[89928]: 2024-03-21 12:58:04.250886949 +0000 UTC m=+0.620622172 container remove 6a632c220c5bb36c8f5eb9ac29ebfd4a785461861354bbb850cda67c8b937c98 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, ceph=True, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, io.openshift.expose-services=, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-21T12:58:04.616 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-21T12:58:04.616 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 systemd[1]: Stopped Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:58:04.616 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.c.service: Consumed 15.445s CPU time. 2024-03-21T12:58:04.616 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 systemd[1]: Starting Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:58:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 podman[90091]: 2024-03-21 12:58:04.736299943 +0000 UTC m=+0.017430548 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:58:04.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:04 smithi172 podman[90091]: 2024-03-21 12:58:04.843952391 +0000 UTC m=+0.125082997 container create 7ee99208840831e894a23a5e8279e33d0df284379739b1384eb73028b98dd1ce (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 podman[90091]: 2024-03-21 12:58:05.088207996 +0000 UTC m=+0.369338612 container init 7ee99208840831e894a23a5e8279e33d0df284379739b1384eb73028b98dd1ce (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 podman[90091]: 2024-03-21 12:58:05.092014977 +0000 UTC m=+0.373145587 container start 7ee99208840831e894a23a5e8279e33d0df284379739b1384eb73028b98dd1ce (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev), process ceph-mon, pid 2 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: pidfile_write: ignore empty --pid-file 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: load: jerasure load: lrc 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: RocksDB version: 7.9.2 2024-03-21T12:58:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Git sha 0 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Compile date 2024-03-21 08:53:26 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: DB SUMMARY 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: DB Session ID: UEJEE99MXRYQB4BTBLQ0 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: CURRENT file: CURRENT 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3209 Bytes 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2794774 ; 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.error_if_exists: 0 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.create_if_missing: 0 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.paranoid_checks: 1 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T12:58:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T12:58:05.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.env: 0x5610f9ad5b20 2024-03-21T12:58:05.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T12:58:05.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.info_log: 0x5610fb67d920 2024-03-21T12:58:05.314 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T12:58:05.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.statistics: (nil) 2024-03-21T12:58:05.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.use_fsync: 0 2024-03-21T12:58:05.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_log_file_size: 0 2024-03-21T12:58:05.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T12:58:05.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T12:58:05.315 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.allow_fallocate: 1 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.use_direct_reads: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.db_log_dir: 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.wal_dir: 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T12:58:05.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.write_buffer_manager: 0x5610fb64e820 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.unordered_write: 0 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T12:58:05.317 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.row_cache: None 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.wal_filter: None 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.two_write_queues: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.wal_compression: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.atomic_flush: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.log_readahead_size: 0 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T12:58:05.318 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_background_jobs: 2 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_background_compactions: -1 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_subcompactions: 1 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T12:58:05.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_open_files: -1 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_background_flushes: -1 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Compression algorithms supported: 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kZSTD supported: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kXpressCompression supported: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kZlibCompression supported: 1 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kSnappyCompression supported: 1 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kLZ4Compression supported: 1 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: kBZip2Compression supported: 0 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T12:58:05.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.merge_operator: 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_filter: None 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5610fb67da20) 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_type: 0 2024-03-21T12:58:05.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_index_type: 0 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_shortening: 1 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: checksum: 4 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: no_block_cache: 0 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache: 0x5610fb681090 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_options: 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: capacity : 536870912 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_shard_bits : 4 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: strict_capacity_limit : 0 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_compressed: (nil) 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: persistent_cache: (nil) 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size: 4096 2024-03-21T12:58:05.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size_deviation: 10 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_restart_interval: 16 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_block_restart_interval: 1 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: metadata_block_size: 4096 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: partition_filters: 0 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: use_delta_encoding: 1 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: filter_policy: bloomfilter 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: whole_key_filtering: 1 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: verify_compression: 0 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: format_version: 5 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_index_compression: 1 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_align: 0 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: max_auto_readahead_size: 262144 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: prepopulate_block_cache: 0 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: initial_auto_readahead_size: 8192 2024-03-21T12:58:05.323 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression: NoCompression 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.num_levels: 7 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T12:58:05.324 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T12:58:05.325 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T12:58:05.326 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.table_properties_collectors: 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.inplace_update_support: 0 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T12:58:05.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.bloom_locality: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.max_successive_merges: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.ttl: 2592000 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.enable_blob_files: false 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.min_blob_size: 0 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T12:58:05.328 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32234, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c64bf87f-3ec8-4c2c-ab05-2eb64893f026 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025885121515, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025885128121, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1779865, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32239, "largest_seqno": 32715, "table_properties": {"data_size": 1776653, "index_size": 1622, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 5273, "raw_average_key_size": 23, "raw_value_size": 1771627, "raw_average_value_size": 8016, "num_data_blocks": 73, "num_entries": 221, "num_filter_entries": 221, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711025885, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c64bf87f-3ec8-4c2c-ab05-2eb64893f026", "db_session_id": "UEJEE99MXRYQB4BTBLQ0", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711025885128240, "job": 1, "event": "recovery_finished"} 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-21T12:58:05.329 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5610fb770000 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: DB pointer 0x5610fb75a000 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** DB Stats ** 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:58:05.330 INFO:journalctl@ceph.mon.c.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: L0 1/0 1.70 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 585.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: L6 1/0 10.56 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: Sum 2/0 12.25 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 585.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 585.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 585.7 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:58:05.331 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative compaction: 0.00 GB write, 190.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval compaction: 0.00 GB write, 190.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: Block cache BinnedLRUCache@0x5610fb681090#2 capacity: 512.00 MB usage: 2.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.66 KB,0.00012517%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: starting mon.c rank 1 at public addrs [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] at bind addrs [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???) e3 preinit fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).mds e1 new map 2024-03-21T12:58:05.332 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).mds e1 print_map 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout: e1 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout: legacy client fscid: -1 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout: No filesystems configured 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 bash[90091]: 7ee99208840831e894a23a5e8279e33d0df284379739b1384eb73028b98dd1ce 2024-03-21T12:58:05.333 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 systemd[1]: Started Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:58:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:05 smithi177 ceph-mon[67966]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:58:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:05 smithi177 ceph-mon[67966]: from='client.25627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:05 smithi177 ceph-mon[67966]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:05 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:05.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:05 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/644351194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:58:05] "GET /metrics HTTP/1.1" 200 38022 "" "Prometheus/2.43.0" 2024-03-21T12:58:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[84635]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:58:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[84635]: from='client.25627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[84635]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:05.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:05.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/644351194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:05.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-21T12:58:05.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: from='client.25627 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:05.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:05.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:05.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:05 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/644351194' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:10.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:58:10.136+0000 7f0d06c0d700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-21T12:58:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:58:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: mon.a calling monitor election 2024-03-21T12:58:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: mon.c calling monitor election 2024-03-21T12:58:10.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: monmap e4: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: fsmap 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: mgrmap e49: y(active, since 22s), standbys: x 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum a,c 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T12:58:10.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: mon.b (rank 2) addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] is down (out of quorum) 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: mon.a calling monitor election 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: mon.c calling monitor election 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:58:10.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: monmap e4: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: fsmap 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: mgrmap e49: y(active, since 22s), standbys: x 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum a,c 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: mon.b (rank 2) addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] is down (out of quorum) 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:10.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:11.434 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:11 smithi172 ceph-mon[84635]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:11.434 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:11.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:11 smithi172 ceph-mon[90110]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:11.435 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T12:58:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: mon.a calling monitor election 2024-03-21T12:58:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: mon.c calling monitor election 2024-03-21T12:58:12.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1006 B/s rd, 0 op/s 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: monmap e4: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: fsmap 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: mgrmap e49: y(active, since 22s), standbys: x 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); 1/3 mons down, quorum a,c 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-21T12:58:12.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: mon.b (rank 2) addr [v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0] is down (out of quorum) 2024-03-21T12:58:12.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:12.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:12.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:12.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:12.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:12 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: mon.b calling monitor election 2024-03-21T12:58:13.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: mon.a calling monitor election 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: mon.c calling monitor election 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: monmap e4: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: fsmap 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: mgrmap e49: y(active, since 24s), standbys: x 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:58:13.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[90110]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: mon.b calling monitor election 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: mon.a calling monitor election 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: mon.c calling monitor election 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: monmap e4: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: fsmap 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: mgrmap e49: y(active, since 24s), standbys: x 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:58:13.648 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:58:13.649 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:13 smithi172 ceph-mon[84635]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: mon.b calling monitor election 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: mon.a calling monitor election 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: mon.c calling monitor election 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: monmap e4: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.177:3300/0,v1:172.21.15.177:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T12:58:13.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: fsmap 2024-03-21T12:58:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T12:58:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: mgrmap e49: y(active, since 24s), standbys: x 2024-03-21T12:58:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-21T12:58:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:58:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:58:13.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:13 smithi177 ceph-mon[67966]: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:58:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:58:15] "GET /metrics HTTP/1.1" 200 38022 "" "Prometheus/2.43.0" 2024-03-21T12:58:15.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-mon[84635]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:15.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:15.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:15.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-mon[90110]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:15.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:15.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:15.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:15 smithi177 ceph-mon[67966]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:15.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:15.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:16.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:17.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:17 smithi172 ceph-mon[84635]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:17.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:17 smithi172 ceph-mon[90110]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:17 smithi177 ceph-mon[67966]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:18.996 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:18.996 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:19.915 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:19.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:19.917 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:20.783 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[90110]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:58:20.783 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[90110]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:20.783 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[90110]: Reconfiguring mon.a (monmap changed)... 2024-03-21T12:58:20.783 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[90110]: Reconfiguring daemon mon.a on smithi172 2024-03-21T12:58:20.784 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[84635]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:58:20.784 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[84635]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:20.784 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[84635]: Reconfiguring mon.a (monmap changed)... 2024-03-21T12:58:20.784 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:20 smithi172 ceph-mon[84635]: Reconfiguring daemon mon.a on smithi172 2024-03-21T12:58:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:20 smithi177 ceph-mon[67966]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:58:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:20 smithi177 ceph-mon[67966]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:20 smithi177 ceph-mon[67966]: Reconfiguring mon.a (monmap changed)... 2024-03-21T12:58:21.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:20 smithi177 ceph-mon[67966]: Reconfiguring daemon mon.a on smithi172 2024-03-21T12:58:22.018 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:22.018 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: Reconfiguring mgr.y (monmap changed)... 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[84635]: Reconfiguring daemon mgr.y on smithi172 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: Reconfiguring mgr.y (monmap changed)... 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:58:22.019 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:58:22.020 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:22.020 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:21 smithi172 ceph-mon[90110]: Reconfiguring daemon mgr.y on smithi172 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: Reconfiguring mgr.y (monmap changed)... 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:21 smithi177 ceph-mon[67966]: Reconfiguring daemon mgr.y on smithi172 2024-03-21T12:58:23.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[84635]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:23.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:23.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[90110]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:23.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:22 smithi177 ceph-mon[67966]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:23.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:23.992 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:23 smithi172 ceph-mon[84635]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:58:23.992 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:23 smithi172 ceph-mon[84635]: Reconfiguring daemon mon.c on smithi172 2024-03-21T12:58:23.992 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:23 smithi172 ceph-mon[90110]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:58:23.992 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:23 smithi172 ceph-mon[90110]: Reconfiguring daemon mon.c on smithi172 2024-03-21T12:58:24.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:23 smithi177 ceph-mon[67966]: Reconfiguring mon.c (monmap changed)... 2024-03-21T12:58:24.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:23 smithi177 ceph-mon[67966]: Reconfiguring daemon mon.c on smithi172 2024-03-21T12:58:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[84635]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-03-21T12:58:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:58:24.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:24.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[90110]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-03-21T12:58:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:58:24.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:24 smithi177 ceph-mon[67966]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 570 B/s rd, 0 op/s 2024-03-21T12:58:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:58:25.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:58:25] "GET /metrics HTTP/1.1" 200 38030 "" "Prometheus/2.43.0" 2024-03-21T12:58:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:25 smithi177 ceph-mon[67966]: Reconfiguring osd.0 (monmap changed)... 2024-03-21T12:58:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:25 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.0 on smithi172 2024-03-21T12:58:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:58:26.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[84635]: Reconfiguring osd.0 (monmap changed)... 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.0 on smithi172 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[90110]: Reconfiguring osd.0 (monmap changed)... 2024-03-21T12:58:26.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.0 on smithi172 2024-03-21T12:58:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:58:26.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:26 smithi177 ceph-mon[67966]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:26 smithi177 ceph-mon[67966]: Reconfiguring osd.1 (monmap changed)... 2024-03-21T12:58:27.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:26 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.1 on smithi172 2024-03-21T12:58:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:26 smithi172 ceph-mon[84635]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:26 smithi172 ceph-mon[84635]: Reconfiguring osd.1 (monmap changed)... 2024-03-21T12:58:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:26 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.1 on smithi172 2024-03-21T12:58:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:26 smithi172 ceph-mon[90110]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:27.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:26 smithi172 ceph-mon[90110]: Reconfiguring osd.1 (monmap changed)... 2024-03-21T12:58:27.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:26 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.1 on smithi172 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: Reconfiguring osd.2 (monmap changed)... 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:28 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.2 on smithi172 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: Reconfiguring osd.2 (monmap changed)... 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.2 on smithi172 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: Reconfiguring osd.2 (monmap changed)... 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:28 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.2 on smithi172 2024-03-21T12:58:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[90110]: Reconfiguring osd.3 (monmap changed)... 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.3 on smithi172 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[84635]: Reconfiguring osd.3 (monmap changed)... 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:29 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.3 on smithi172 2024-03-21T12:58:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:29 smithi177 ceph-mon[67966]: Reconfiguring osd.3 (monmap changed)... 2024-03-21T12:58:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T12:58:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:29 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.3 on smithi172 2024-03-21T12:58:31.218 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[84635]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[90110]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:31.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:30 smithi177 ceph-mon[67966]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:58:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:32.203 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:32.203 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:31 smithi172 ceph-mon[84635]: Reconfiguring rgw.foo.smithi172.aecdpn (monmap changed)... 2024-03-21T12:58:32.203 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:31 smithi172 ceph-mon[84635]: Reconfiguring daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T12:58:32.204 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:32.204 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:31 smithi172 ceph-mon[90110]: Reconfiguring rgw.foo.smithi172.aecdpn (monmap changed)... 2024-03-21T12:58:32.204 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:31 smithi172 ceph-mon[90110]: Reconfiguring daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T12:58:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:31 smithi177 ceph-mon[67966]: Reconfiguring rgw.foo.smithi172.aecdpn (monmap changed)... 2024-03-21T12:58:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:31 smithi177 ceph-mon[67966]: Reconfiguring daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T12:58:33.121 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[84635]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:33.121 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:33.121 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[90110]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:33.122 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:32 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:32 smithi177 ceph-mon[67966]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:32 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:32 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:32 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:32 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:34.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:34.157 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:34.157 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:34.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:34.158 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:34.158 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:34.158 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:58:33.994+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:34.158 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:34.158 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:34.158 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:34.159 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:58:34.160 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:34.161 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:34.162 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:34.163 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:33 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:33 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:34.399 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:58:35.046 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (3m) 20s ago 25m 17.8M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (2m) 42s ago 24m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 20s ago 24m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (4m) 42s ago 29m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (14m) 20s ago 31m 495M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (57s) 20s ago 31m 40.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (108s) 42s ago 29m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (30s) 20s ago 30m 20.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (3m) 20s ago 25m 8879k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (3m) 42s ago 25m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (29m) 20s ago 29m 57.1M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (28m) 20s ago 28m 57.9M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (28m) 20s ago 28m 49.3M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (28m) 20s ago 28m 52.6M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:58:35.047 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (27m) 42s ago 27m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:58:35.048 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (27m) 42s ago 27m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:58:35.048 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (26m) 42s ago 26m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:58:35.048 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (26m) 42s ago 26m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:58:35.048 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (3m) 42s ago 25m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:58:35.048 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (24m) 20s ago 24m 77.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:58:35.048 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (24m) 42s ago 24m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:35.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:35.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:35.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:35.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:35.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:34 smithi177 ceph-mon[67966]: Reconfiguring daemon mon.b on smithi177 2024-03-21T12:58:35.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:35.312 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.314 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:35.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:35.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:58:35.317 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:35.318 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:35.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[84635]: Reconfiguring daemon mon.b on smithi177 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: Reconfiguring mon.b (monmap changed)... 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:35.320 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:34 smithi172 ceph-mon[90110]: Reconfiguring daemon mon.b on smithi177 2024-03-21T12:58:35.440 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:58:35.441 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-21T12:58:35.442 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 5 2024-03-21T12:58:35.442 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:58:35.442 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:58:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:58:35] "GET /metrics HTTP/1.1" 200 38031 "" "Prometheus/2.43.0" 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi172", 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "mon" 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "2/2 daemons upgraded", 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading mon daemons", 2024-03-21T12:58:35.784 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:58:35.785 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:58:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:36.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2579889776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2579889776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:58:36.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:58:36.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:36.249 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='client.64108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2579889776' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T12:58:36.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:36 smithi177 ceph-mon[67966]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:36 smithi177 ceph-mon[67966]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:36 smithi177 ceph-mon[67966]: Reconfiguring mgr.x (monmap changed)... 2024-03-21T12:58:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:36 smithi177 ceph-mon[67966]: Reconfiguring daemon mgr.x on smithi177 2024-03-21T12:58:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:36 smithi177 ceph-mon[67966]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[90110]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[90110]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[90110]: Reconfiguring mgr.x (monmap changed)... 2024-03-21T12:58:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[90110]: Reconfiguring daemon mgr.x on smithi177 2024-03-21T12:58:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[90110]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[84635]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:37.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[84635]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:37.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[84635]: Reconfiguring mgr.x (monmap changed)... 2024-03-21T12:58:37.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[84635]: Reconfiguring daemon mgr.x on smithi177 2024-03-21T12:58:37.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:36 smithi172 ceph-mon[84635]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: Reconfiguring osd.4 (monmap changed)... 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:38 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.4 on smithi177 2024-03-21T12:58:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:38.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: Reconfiguring osd.4 (monmap changed)... 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.4 on smithi177 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: Reconfiguring osd.4 (monmap changed)... 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:38.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:38 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.4 on smithi177 2024-03-21T12:58:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:39 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:39 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:39 smithi177 ceph-mon[67966]: Reconfiguring osd.5 (monmap changed)... 2024-03-21T12:58:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:39 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:58:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:39 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:39 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.5 on smithi177 2024-03-21T12:58:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[84635]: Reconfiguring osd.5 (monmap changed)... 2024-03-21T12:58:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.5 on smithi177 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[90110]: Reconfiguring osd.5 (monmap changed)... 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:39 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.5 on smithi177 2024-03-21T12:58:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:40 smithi177 ceph-mon[67966]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:40 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:40 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:40 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:58:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:40 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[90110]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:58:41.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[84635]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T12:58:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:40 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:42.096 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:41 smithi177 ceph-mon[67966]: Reconfiguring osd.6 (monmap changed)... 2024-03-21T12:58:42.096 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:41 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.6 on smithi177 2024-03-21T12:58:42.096 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:41 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:41 smithi172 ceph-mon[84635]: Reconfiguring osd.6 (monmap changed)... 2024-03-21T12:58:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:41 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.6 on smithi177 2024-03-21T12:58:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:41 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:41 smithi172 ceph-mon[90110]: Reconfiguring osd.6 (monmap changed)... 2024-03-21T12:58:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:41 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.6 on smithi177 2024-03-21T12:58:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:41 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:43.065 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:42 smithi177 ceph-mon[67966]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:43.065 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:43.065 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:43.065 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:58:43.066 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[90110]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:58:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[84635]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T12:58:43.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:43 smithi177 ceph-mon[67966]: Reconfiguring osd.7 (monmap changed)... 2024-03-21T12:58:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:43 smithi177 ceph-mon[67966]: Reconfiguring daemon osd.7 on smithi177 2024-03-21T12:58:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:43 smithi172 ceph-mon[84635]: Reconfiguring osd.7 (monmap changed)... 2024-03-21T12:58:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:43 smithi172 ceph-mon[84635]: Reconfiguring daemon osd.7 on smithi177 2024-03-21T12:58:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:43 smithi172 ceph-mon[90110]: Reconfiguring osd.7 (monmap changed)... 2024-03-21T12:58:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:43 smithi172 ceph-mon[90110]: Reconfiguring daemon osd.7 on smithi177 2024-03-21T12:58:45.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:44 smithi177 ceph-mon[67966]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:45.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:45.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:45.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:45.250 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[84635]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[90110]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:58:45.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:45.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:45.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T12:58:45.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:58:45] "GET /metrics HTTP/1.1" 200 38031 "" "Prometheus/2.43.0" 2024-03-21T12:58:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: Reconfiguring rgw.foo.smithi177.magnlc (monmap changed)... 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: Reconfiguring daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:58:46.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:46.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:46.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:45 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: Reconfiguring rgw.foo.smithi177.magnlc (monmap changed)... 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: Reconfiguring daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: Reconfiguring rgw.foo.smithi177.magnlc (monmap changed)... 2024-03-21T12:58:46.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: Reconfiguring daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.399 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:58:46.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T12:58:46.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:58:46.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:46.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T12:58:46.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:46.407 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:46.408 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:46.408 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:46.408 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:46.408 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:45 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mon 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all crash 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mds 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nfs 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all node-exporter 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all prometheus 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all alertmanager 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all grafana 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all loki 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all promtail 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Finalizing container_image settings 2024-03-21T12:58:47.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Upgrade: Complete! 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mon 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all crash 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mds 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nfs 2024-03-21T12:58:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all node-exporter 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all prometheus 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all alertmanager 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all grafana 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all loki 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all promtail 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Finalizing container_image settings 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Upgrade: Complete! 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:47.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:46 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mon 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mds 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nfs 2024-03-21T12:58:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nvmeof 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all node-exporter 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all prometheus 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all alertmanager 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all grafana 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all loki 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all promtail 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Finalizing container_image settings 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Upgrade: Complete! 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 861 B/s rd, 0 op/s 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:47.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:46 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:58:47.657+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:47.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:47.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:58:47.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:47.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:47.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:48.369 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:58:48.370 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:58:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:49.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 477 B/s rd, 0 op/s 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 0 op/s 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:49.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.392 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.393 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:49.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 477 B/s rd, 0 op/s 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 0 op/s 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:49.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:49.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 477 B/s rd, 0 op/s 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 623 B/s rd, 0 op/s 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:58:49.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:58:49.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:58:49.538+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:49.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:58:49.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:49.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:50 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:51.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:51.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:51.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:50 smithi177 ceph-mon[67966]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:58:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:50 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:51.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:51.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:50 smithi172 ceph-mon[90110]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:50 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:51.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:58:51.402 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:58:51.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:58:51.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:58:51.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:50 smithi172 ceph-mon[84635]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T12:58:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:58:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:52 smithi177 ceph-mon[67966]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 778 B/s rd, 0 op/s 2024-03-21T12:58:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:52 smithi172 ceph-mon[84635]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 778 B/s rd, 0 op/s 2024-03-21T12:58:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:52 smithi172 ceph-mon[90110]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 778 B/s rd, 0 op/s 2024-03-21T12:58:55.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:54 smithi172 ceph-mon[84635]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-21T12:58:55.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:54 smithi172 ceph-mon[90110]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-21T12:58:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:54 smithi177 ceph-mon[67966]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-03-21T12:58:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:58:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:58:55] "GET /metrics HTTP/1.1" 200 38031 "" "Prometheus/2.43.0" 2024-03-21T12:58:57.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:56 smithi177 ceph-mon[67966]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:58:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:56 smithi172 ceph-mon[84635]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:58:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:56 smithi172 ceph-mon[90110]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T12:58:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:58:58 smithi177 ceph-mon[67966]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:58:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:58:58 smithi172 ceph-mon[84635]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:58:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:58:58 smithi172 ceph-mon[90110]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:00 smithi177 ceph-mon[67966]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:00 smithi172 ceph-mon[84635]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:00 smithi172 ceph-mon[90110]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:02.319 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:02 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:02.319 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:02 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:02.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:02 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:03.321 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:03 smithi177 ceph-mon[67966]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:03.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:03 smithi172 ceph-mon[84635]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:03 smithi172 ceph-mon[90110]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:04 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:04 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:05.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:05 smithi172 ceph-mon[84635]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:05.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:05 smithi172 ceph-mon[90110]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:05 smithi177 ceph-mon[67966]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:59:05] "GET /metrics HTTP/1.1" 200 38029 "" "Prometheus/2.43.0" 2024-03-21T12:59:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:06 smithi177 ceph-mon[67966]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:06 smithi172 ceph-mon[84635]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:06.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:06 smithi172 ceph-mon[90110]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:06.840 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T12:59:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:07 smithi172 ceph-mon[84635]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:07.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:07 smithi172 ceph-mon[90110]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:07 smithi177 ceph-mon[67966]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:08.119 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:08 smithi172 ceph-mon[90110]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:08.119 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:08 smithi172 ceph-mon[84635]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:08 smithi177 ceph-mon[67966]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:08.872 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (3m) 54s ago 25m 17.8M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (3m) 76s ago 25m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 54s ago 24m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (4m) 76s ago 30m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (14m) 54s ago 31m 495M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (91s) 54s ago 31m 40.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (2m) 76s ago 30m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (64s) 54s ago 30m 20.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (4m) 54s ago 26m 8879k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (4m) 76s ago 26m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (29m) 54s ago 29m 57.1M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (29m) 54s ago 29m 57.9M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:59:08.873 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (29m) 54s ago 29m 49.3M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (28m) 54s ago 28m 52.6M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (28m) 76s ago 28m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (27m) 76s ago 27m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (27m) 76s ago 27m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (27m) 76s ago 27m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (4m) 76s ago 26m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (24m) 54s ago 24m 77.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:59:08.874 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (24m) 76s ago 24m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:59:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:09 smithi172 ceph-mon[84635]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:09 smithi172 ceph-mon[90110]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:09 smithi177 ceph-mon[67966]: from='client.54204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:09.516 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-03-21T12:59:10.122 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:10 smithi172 ceph-mon[84635]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:10.122 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:10 smithi172 ceph-mon[90110]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:10 smithi177 ceph-mon[67966]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:11.129 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:11.129 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:11.535 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:59:12.191 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:12 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2897071974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:12.191 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:12 smithi172 ceph-mon[84635]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:12.191 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:12 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2897071974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:12.191 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:12 smithi172 ceph-mon[90110]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:12.215 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-03-21T12:59:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:12 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2897071974' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:12 smithi177 ceph-mon[67966]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:14.358 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)" 2024-03-21T12:59:14.965 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:14 smithi172 ceph-mon[84635]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:14.965 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:14 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/45212648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:14.965 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:14 smithi172 ceph-mon[90110]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:14.965 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:14 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/45212648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:14.989 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-03-21T12:59:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:14 smithi177 ceph-mon[67966]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:15.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:14 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/45212648' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:15.617 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:59:15] "GET /metrics HTTP/1.1" 200 38029 "" "Prometheus/2.43.0" 2024-03-21T12:59:16.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:16 smithi172 ceph-mon[84635]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:16.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:16 smithi172 ceph-mon[90110]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:16 smithi177 ceph-mon[67966]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:17.990 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:17 smithi172 ceph-mon[84635]: from='client.54222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:17.990 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:17 smithi172 ceph-mon[90110]: from='client.54222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:17 smithi177 ceph-mon[67966]: from='client.54222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:19.003 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:59:19.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:18 smithi172 ceph-mon[84635]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:19.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:19.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:18 smithi172 ceph-mon[90110]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:19.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:18 smithi177 ceph-mon[67966]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:19.670 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T12:59:21.190 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:20 smithi172 ceph-mon[84635]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:21.191 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:20 smithi172 ceph-mon[90110]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:20 smithi177 ceph-mon[67966]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:21.593 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:59:21.593 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T12:59:21.593 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T12:59:21.593 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T12:59:21.594 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:59:21.594 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T12:59:21.594 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T12:59:21.594 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:59:21.594 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:59:22.196 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:21 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:22.196 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:21 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:22.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T12:59:22.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:21 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:23.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:22 smithi172 ceph-mon[84635]: from='client.64162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:23.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:22 smithi172 ceph-mon[84635]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:23.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:22 smithi172 ceph-mon[90110]: from='client.64162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:23.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:22 smithi172 ceph-mon[90110]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:23.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:22 smithi177 ceph-mon[67966]: from='client.64162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:22 smithi177 ceph-mon[67966]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:24.315 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T12:59:24.315 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T12:59:24.315 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T12:59:24.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-03-21T12:59:25.258 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:24 smithi172 ceph-mon[84635]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:25.258 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:24 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3429503291' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:59:25.258 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:24 smithi172 ceph-mon[90110]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:25.258 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:24 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3429503291' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:59:25.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:24 smithi177 ceph-mon[67966]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:25.400 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:24 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3429503291' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T12:59:25.563 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:59:25] "GET /metrics HTTP/1.1" 200 38028 "" "Prometheus/2.43.0" 2024-03-21T12:59:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:26 smithi177 ceph-mon[67966]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:26 smithi172 ceph-mon[84635]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:26 smithi172 ceph-mon[90110]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:28.106 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:27 smithi172 ceph-mon[84635]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:28.107 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:27 smithi172 ceph-mon[90110]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:28.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:27 smithi177 ceph-mon[67966]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:28.946 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:29.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:28 smithi172 ceph-mon[84635]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:29.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:28 smithi172 ceph-mon[90110]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:29.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:28 smithi177 ceph-mon[67966]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T12:59:29.521 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T12:59:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:30.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[84635]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[90110]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:31.113 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:31.114 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:31.114 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:31.114 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:59:31.114 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:31.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:30 smithi177 ceph-mon[67966]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T12:59:31.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:31.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:31.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:31.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:59:31.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:31.557 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T12:59:32.227 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T12:59:32.227 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (4m) 77s ago 26m 17.8M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T12:59:32.227 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (3m) 99s ago 25m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 77s ago 25m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (5m) 99s ago 30m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (15m) 77s ago 32m 495M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (114s) 77s ago 32m 40.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (2m) 99s ago 30m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (87s) 77s ago 30m 20.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (4m) 77s ago 26m 8879k - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (4m) 99s ago 26m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (30m) 77s ago 30m 57.1M 2992M 17.2.0 e1d6a67b021e dcd2e5daba9a 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (29m) 77s ago 29m 57.9M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (29m) 77s ago 29m 49.3M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (29m) 77s ago 29m 52.6M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (28m) 99s ago 28m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T12:59:32.228 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (28m) 99s ago 28m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T12:59:32.229 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (27m) 99s ago 27m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T12:59:32.229 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (27m) 99s ago 27m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T12:59:32.229 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (4m) 99s ago 26m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T12:59:32.229 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (25m) 77s ago 25m 77.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T12:59:32.229 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (25m) 99s ago 25m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T12:59:32.229 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:32.229 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[84635]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:59:32.229 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[84635]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:32.229 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:59:32.229 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:59:32.230 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:32.230 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[90110]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:59:32.230 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[90110]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:32.230 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:59:32.230 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:31 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:59:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:31 smithi177 ceph-mon[67966]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 op/s 2024-03-21T12:59:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:31 smithi177 ceph-mon[67966]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:31 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:59:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:31 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T12:59:32.638 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T12:59:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T12:59:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T12:59:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-21T12:59:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 5 2024-03-21T12:59:32.639 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T12:59:32.639 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:59:32.893 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:59:32.551+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:32.893 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:59:32.893 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:32.893 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:32.894 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:32.895 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:32.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:32.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:32.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:32.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:32.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:32.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:32.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:32.984 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T12:59:32.984 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", 2024-03-21T12:59:32.984 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T12:59:32.984 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-03-21T12:59:32.984 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T12:59:32.984 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "", 2024-03-21T12:59:32.985 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 image", 2024-03-21T12:59:32.985 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T12:59:32.985 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T12:59:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-mon[84635]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-mon[84635]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/698346188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:33.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-mon[90110]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-mon[90110]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:32 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/698346188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:32 smithi177 ceph-mon[67966]: from='client.64180 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:32 smithi177 ceph-mon[67966]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:32 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/698346188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:34.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[90110]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:34.148 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:34.149 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:34.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[84635]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:34.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:34.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:34.153 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:34.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[84635]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[84635]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[84635]: from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:34.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[90110]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[90110]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[90110]: from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:34.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:33 smithi177 ceph-mon[67966]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:33 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:34.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:34.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:34.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:33 smithi177 ceph-mon[67966]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:33 smithi177 ceph-mon[67966]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:34.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:33 smithi177 ceph-mon[67966]: from='client.64204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T12:59:34.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:35.598 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:59:35] "GET /metrics HTTP/1.1" 200 38025 "" "Prometheus/2.43.0" 2024-03-21T12:59:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mgr 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mon 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all crash 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T12:59:35.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[90110]: Upgrade: osd.0 is safe to restart 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mgr 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mon 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:35.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all crash 2024-03-21T12:59:35.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:35.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T12:59:35.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T12:59:35.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:35 smithi172 ceph-mon[84635]: Upgrade: osd.0 is safe to restart 2024-03-21T12:59:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T12:59:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T12:59:36.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mgr 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mon 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-21T12:59:36.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:35 smithi177 ceph-mon[67966]: Upgrade: osd.0 is safe to restart 2024-03-21T12:59:37.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[84635]: Upgrade: Updating osd.0 2024-03-21T12:59:37.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:37.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:59:37.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[84635]: Deploying daemon osd.0 on smithi172 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[84635]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[90110]: Upgrade: Updating osd.0 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[90110]: Deploying daemon osd.0 on smithi172 2024-03-21T12:59:37.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:37 smithi172 ceph-mon[90110]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:37 smithi177 ceph-mon[67966]: Upgrade: Updating osd.0 2024-03-21T12:59:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:37 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:37 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T12:59:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:37 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:37 smithi177 ceph-mon[67966]: Deploying daemon osd.0 on smithi172 2024-03-21T12:59:37.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:37 smithi177 ceph-mon[67966]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:39.946 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:39 smithi172 ceph-mon[84635]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-21T12:59:39.947 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:39 smithi172 ceph-mon[90110]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-21T12:59:39.947 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:39 smithi172 systemd[1]: Stopping Ceph osd.0 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:59:40.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:39 smithi177 ceph-mon[67966]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 573 B/s rd, 0 op/s 2024-03-21T12:59:40.396 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[32073]: 2024-03-21T12:59:39.944+0000 7f33e917c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T12:59:40.396 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[32073]: 2024-03-21T12:59:39.944+0000 7f33e917c700 -1 osd.0 106 *** Got signal Terminated *** 2024-03-21T12:59:40.396 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:39 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[32073]: 2024-03-21T12:59:39.944+0000 7f33e917c700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T12:59:40.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:40 smithi172 ceph-mon[84635]: osd.0 marked itself down and dead 2024-03-21T12:59:40.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:40 smithi172 ceph-mon[90110]: osd.0 marked itself down and dead 2024-03-21T12:59:41.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:40 smithi177 ceph-mon[67966]: osd.0 marked itself down and dead 2024-03-21T12:59:41.429 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:41 smithi172 podman[97884]: 2024-03-21 12:59:41.173046857 +0000 UTC m=+1.322811147 container died dcd2e5daba9ae672ee14c9933a86d5a6cf89d46b6bcb06d1f4805e07f87c8382 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream) 2024-03-21T12:59:41.713 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:41.713 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[84635]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:41.713 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T12:59:41.713 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[84635]: osdmap e107: 8 total, 7 up, 8 in 2024-03-21T12:59:41.713 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:41.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[90110]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:41.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T12:59:41.714 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:41 smithi172 ceph-mon[90110]: osdmap e107: 8 total, 7 up, 8 in 2024-03-21T12:59:41.714 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:41 smithi172 podman[97884]: 2024-03-21 12:59:41.520587371 +0000 UTC m=+1.670351661 container cleanup dcd2e5daba9ae672ee14c9933a86d5a6cf89d46b6bcb06d1f4805e07f87c8382 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, GIT_BRANCH=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-21T12:59:41.714 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:41 smithi172 bash[97884]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0 2024-03-21T12:59:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:41 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:41 smithi177 ceph-mon[67966]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T12:59:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:41 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T12:59:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:41 smithi177 ceph-mon[67966]: osdmap e107: 8 total, 7 up, 8 in 2024-03-21T12:59:42.132 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:41 smithi172 podman[97901]: 2024-03-21 12:59:41.713365202 +0000 UTC m=+0.537833716 container remove dcd2e5daba9ae672ee14c9933a86d5a6cf89d46b6bcb06d1f4805e07f87c8382 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, RELEASE=HEAD, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, release=754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., distribution-scope=public, ceph=True, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-03-21T12:59:42.396 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:42 smithi172 podman[97992]: 2024-03-21 12:59:42.035389704 +0000 UTC m=+0.018384284 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:59:42.396 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:42 smithi172 podman[97992]: 2024-03-21 12:59:42.164742454 +0000 UTC m=+0.147737075 container create 88c0c0a7c5e4d845b206492fcdd00c233a6840a98fdec16e1ac557da6bfdf9e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:59:42.665 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:42 smithi172 podman[97992]: 2024-03-21 12:59:42.41015088 +0000 UTC m=+0.393145466 container init 88c0c0a7c5e4d845b206492fcdd00c233a6840a98fdec16e1ac557da6bfdf9e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T12:59:42.666 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:42 smithi172 podman[97992]: 2024-03-21 12:59:42.413443964 +0000 UTC m=+0.396438547 container start 88c0c0a7c5e4d845b206492fcdd00c233a6840a98fdec16e1ac557da6bfdf9e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T12:59:42.666 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:42 smithi172 podman[97992]: 2024-03-21 12:59:42.492255055 +0000 UTC m=+0.475249645 container attach 88c0c0a7c5e4d845b206492fcdd00c233a6840a98fdec16e1ac557da6bfdf9e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T12:59:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:42 smithi177 ceph-mon[67966]: osdmap e108: 8 total, 7 up, 8 in 2024-03-21T12:59:43.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:42 smithi172 ceph-mon[84635]: osdmap e108: 8 total, 7 up, 8 in 2024-03-21T12:59:43.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:42 smithi172 ceph-mon[90110]: osdmap e108: 8 total, 7 up, 8 in 2024-03-21T12:59:43.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:42 smithi172 podman[97992]: 2024-03-21 12:59:42.711069436 +0000 UTC m=+0.694064023 container died 88c0c0a7c5e4d845b206492fcdd00c233a6840a98fdec16e1ac557da6bfdf9e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-21T12:59:43.424 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 podman[98036]: 2024-03-21 12:59:43.41235415 +0000 UTC m=+0.696448297 container remove 88c0c0a7c5e4d845b206492fcdd00c233a6840a98fdec16e1ac557da6bfdf9e1 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-21T12:59:43.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:43 smithi172 ceph-mon[84635]: pgmap v69: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:43.854 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:43 smithi172 ceph-mon[90110]: pgmap v69: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:43.855 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-21T12:59:43.855 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 systemd[1]: Stopped Ceph osd.0 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:59:43.855 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.0.service: Consumed 14.097s CPU time. 2024-03-21T12:59:43.855 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 systemd[1]: Starting Ceph osd.0 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T12:59:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:43 smithi177 ceph-mon[67966]: pgmap v69: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T12:59:44.146 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 podman[98139]: 2024-03-21 12:59:43.854912625 +0000 UTC m=+0.114509849 container create e6770b87fc3967ec0908e5b9aa47c5a9fd4b7c912f667ddb77a7c1bf00ef98da (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-21T12:59:44.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:43 smithi172 podman[98139]: 2024-03-21 12:59:43.758470444 +0000 UTC m=+0.018067668 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:59:44.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 podman[98139]: 2024-03-21 12:59:44.09099721 +0000 UTC m=+0.350594444 container init e6770b87fc3967ec0908e5b9aa47c5a9fd4b7c912f667ddb77a7c1bf00ef98da (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-21T12:59:44.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 podman[98139]: 2024-03-21 12:59:44.096168693 +0000 UTC m=+0.355765924 container start e6770b87fc3967ec0908e5b9aa47c5a9fd4b7c912f667ddb77a7c1bf00ef98da (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-21T12:59:44.572 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 podman[98139]: 2024-03-21 12:59:44.184899471 +0000 UTC m=+0.444496704 container attach e6770b87fc3967ec0908e5b9aa47c5a9fd4b7c912f667ddb77a7c1bf00ef98da (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-21T12:59:44.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T12:59:44.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 bash[98139]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T12:59:44.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-21T12:59:44.898 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-21T12:59:44.898 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate[98154]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-21T12:59:44.898 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:44 smithi172 podman[98139]: 2024-03-21 12:59:44.597595814 +0000 UTC m=+0.857193039 container died e6770b87fc3967ec0908e5b9aa47c5a9fd4b7c912f667ddb77a7c1bf00ef98da (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T12:59:45.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:44 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T12:59:45.585 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:59:45] "GET /metrics HTTP/1.1" 200 38025 "" "Prometheus/2.43.0" 2024-03-21T12:59:45.585 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 podman[98289]: 2024-03-21 12:59:45.236643261 +0000 UTC m=+0.630387543 container remove e6770b87fc3967ec0908e5b9aa47c5a9fd4b7c912f667ddb77a7c1bf00ef98da (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-21T12:59:45.585 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 podman[98352]: 2024-03-21 12:59:45.351666062 +0000 UTC m=+0.018126595 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T12:59:45.585 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 podman[98352]: 2024-03-21 12:59:45.454139134 +0000 UTC m=+0.120599705 container create bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T12:59:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:45 smithi177 ceph-mon[67966]: pgmap v70: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:45 smithi172 ceph-mon[84635]: pgmap v70: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:45 smithi172 ceph-mon[90110]: pgmap v70: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:46.146 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 podman[98352]: 2024-03-21 12:59:45.698566586 +0000 UTC m=+0.365027115 container init bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-21T12:59:46.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 podman[98352]: 2024-03-21 12:59:45.701884988 +0000 UTC m=+0.368345516 container start bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T12:59:46.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 ceph-osd[98371]: -- 172.21.15.172:0/2271589540 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a709381680 con 0x55a70934ac00 2024-03-21T12:59:46.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 bash[98352]: bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 2024-03-21T12:59:46.147 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:45 smithi172 systemd[1]: Started Ceph osd.0 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T12:59:46.646 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T12:59:46.250+0000 7f5366ecc7c0 -1 Falling back to public interface 2024-03-21T12:59:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:46 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:46 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:46 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:46 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:46 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:47 smithi177 ceph-mon[67966]: pgmap v71: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:48.364 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:47 smithi172 ceph-mon[84635]: pgmap v71: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:48.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:47 smithi172 ceph-mon[90110]: pgmap v71: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:49.311 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:48 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T12:59:48.982+0000 7f5366ecc7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-21T12:59:49.311 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T12:59:49.016+0000 7f5366ecc7c0 -1 osd.0 106 log_to_monitors true 2024-03-21T12:59:49.311 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:49.311 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:49.311 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[84635]: pgmap v72: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[84635]: from='osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[84635]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[90110]: pgmap v72: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[90110]: from='osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:59:49.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-mon[90110]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:59:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T12:59:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:49 smithi177 ceph-mon[67966]: pgmap v72: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 65/627 objects degraded (10.367%) 2024-03-21T12:59:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:49 smithi177 ceph-mon[67966]: from='osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:59:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:49 smithi177 ceph-mon[67966]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T12:59:49.646 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 12:59:49 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T12:59:49.504+0000 7f535431c700 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T12:59:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:50 smithi177 ceph-mon[67966]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:59:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:50 smithi177 ceph-mon[67966]: osdmap e109: 8 total, 7 up, 8 in 2024-03-21T12:59:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:50 smithi177 ceph-mon[67966]: from='osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:59:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:50 smithi177 ceph-mon[67966]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:59:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:50 smithi177 ceph-mon[67966]: from='osd.0 ' entity='osd.0' 2024-03-21T12:59:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[90110]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:59:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[90110]: osdmap e109: 8 total, 7 up, 8 in 2024-03-21T12:59:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[90110]: from='osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:59:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[90110]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:59:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[90110]: from='osd.0 ' entity='osd.0' 2024-03-21T12:59:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[84635]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T12:59:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[84635]: osdmap e109: 8 total, 7 up, 8 in 2024-03-21T12:59:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[84635]: from='osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:59:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[84635]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T12:59:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:50 smithi172 ceph-mon[84635]: from='osd.0 ' entity='osd.0' 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406] boot 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:51.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:51 smithi177 ceph-mon[67966]: pgmap v75: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406] boot 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:51.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[90110]: pgmap v75: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: osd.0 [v2:172.21.15.172:6802/1525817406,v1:172.21.15.172:6803/1525817406] boot 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T12:59:51.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:51 smithi172 ceph-mon[84635]: pgmap v75: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:52 smithi172 ceph-mon[84635]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T12:59:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:52.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:52 smithi172 ceph-mon[90110]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T12:59:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:52 smithi177 ceph-mon[67966]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T12:59:53.560 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:53 smithi172 ceph-mon[84635]: pgmap v77: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:53.561 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:53 smithi172 ceph-mon[90110]: pgmap v77: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:53.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:53 smithi177 ceph-mon[67966]: pgmap v77: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 113 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 65/627 objects degraded (10.367%) 2024-03-21T12:59:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:54 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded) 2024-03-21T12:59:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:54 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded) 2024-03-21T12:59:55.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:54 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 65/627 objects degraded (10.367%), 19 pgs degraded) 2024-03-21T12:59:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:12:59:55] "GET /metrics HTTP/1.1" 200 38161 "" "Prometheus/2.43.0" 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:59:56.001 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:59:56.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T12:59:56.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:56.972 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[84635]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-03-21T12:59:56.972 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[84635]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:56.972 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:59:56.972 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:59:56.972 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[90110]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-03-21T12:59:56.972 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[90110]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:56.973 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:59:56.973 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:59:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:56 smithi177 ceph-mon[67966]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 848 B/s rd, 0 op/s 2024-03-21T12:59:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:56 smithi177 ceph-mon[67966]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T12:59:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:56 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T12:59:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:56 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T12:59:57.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T12:59:56.971+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:57.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T12:59:57.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:57.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 12:59:56 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.834 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:57.835 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:57.836 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:57.837 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:57.838 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:57.839 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:57.840 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:57.841 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[90110]: Upgrade: osd.1 is safe to restart 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:57.842 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T12:59:57.843 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:57 smithi172 ceph-mon[84635]: Upgrade: osd.1 is safe to restart 2024-03-21T12:59:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:58.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T12:59:58.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T12:59:58.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-21T12:59:58.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:57 smithi177 ceph-mon[67966]: Upgrade: osd.1 is safe to restart 2024-03-21T12:59:59.274 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[84635]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[90110]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:59:59.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:58 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T12:59:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:58 smithi177 ceph-mon[67966]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T12:59:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T12:59:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T12:59:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:58 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:00.185 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:59 smithi172 ceph-mon[84635]: Upgrade: Updating osd.1 2024-03-21T13:00:00.186 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 12:59:59 smithi172 ceph-mon[84635]: Deploying daemon osd.1 on smithi172 2024-03-21T13:00:00.186 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:59 smithi172 ceph-mon[90110]: Upgrade: Updating osd.1 2024-03-21T13:00:00.186 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 12:59:59 smithi172 ceph-mon[90110]: Deploying daemon osd.1 on smithi172 2024-03-21T13:00:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:59 smithi177 ceph-mon[67966]: Upgrade: Updating osd.1 2024-03-21T13:00:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 12:59:59 smithi177 ceph-mon[67966]: Deploying daemon osd.1 on smithi172 2024-03-21T13:00:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:00 smithi172 ceph-mon[84635]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:00 smithi172 ceph-mon[84635]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:00:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:00 smithi172 ceph-mon[90110]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:01.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:00 smithi172 ceph-mon[90110]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:00:01.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:00 smithi177 ceph-mon[67966]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:01.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:00 smithi177 ceph-mon[67966]: overall HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:00:02.186 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:02.187 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:01 smithi172 systemd[1]: Stopping Ceph osd.1 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:00:02.187 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:01 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:02.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[35034]: 2024-03-21T13:00:02.230+0000 7f6e191d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:00:02.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[35034]: 2024-03-21T13:00:02.230+0000 7f6e191d4700 -1 osd.1 111 *** Got signal Terminated *** 2024-03-21T13:00:02.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[35034]: 2024-03-21T13:00:02.230+0000 7f6e191d4700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:00:03.300 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-mon[84635]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:03.300 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-mon[84635]: osd.1 marked itself down and dead 2024-03-21T13:00:03.301 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-mon[90110]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:03.301 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:02 smithi172 ceph-mon[90110]: osd.1 marked itself down and dead 2024-03-21T13:00:03.357 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:00:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:02 smithi177 ceph-mon[67966]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:02 smithi177 ceph-mon[67966]: osd.1 marked itself down and dead 2024-03-21T13:00:03.590 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:03 smithi172 podman[102647]: 2024-03-21 13:00:03.300185669 +0000 UTC m=+1.193617149 container died 3ea41cf71314da6d8cabff77a146eb9ce04cec489dcc4597aaeae9fbf39e9cec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, version=8, CEPH_POINT_RELEASE=-17.2.0, release=754, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-21T13:00:03.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:03 smithi172 podman[102647]: 2024-03-21 13:00:03.664296976 +0000 UTC m=+1.557728459 container cleanup 3ea41cf71314da6d8cabff77a146eb9ce04cec489dcc4597aaeae9fbf39e9cec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, RELEASE=HEAD, ceph=True, vcs-type=git, io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , architecture=x86_64, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True) 2024-03-21T13:00:03.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:03 smithi172 bash[102647]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1 2024-03-21T13:00:04.054 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:00:04.054 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (4m) 13s ago 26m 26.8M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (4m) 2m ago 26m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 13s ago 25m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (5m) 2m ago 31m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (15m) 13s ago 32m 504M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (2m) 13s ago 32m 45.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (3m) 2m ago 31m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (119s) 13s ago 31m 37.2M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (5m) 13s ago 27m 16.4M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (5m) 2m ago 27m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (18s) 13s ago 30m 12.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (30m) 13s ago 30m 58.7M 2992M 17.2.0 e1d6a67b021e 3ea41cf71314 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (29m) 13s ago 29m 50.5M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (29m) 13s ago 29m 53.2M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (29m) 2m ago 29m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T13:00:04.055 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (28m) 2m ago 28m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:00:04.056 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (28m) 2m ago 28m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:00:04.056 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (28m) 2m ago 28m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:00:04.056 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (4m) 2m ago 27m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:00:04.056 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (25m) 13s ago 25m 77.4M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:00:04.056 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (25m) 2m ago 25m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:00:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:03 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:00:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:03 smithi177 ceph-mon[67966]: osdmap e112: 8 total, 7 up, 8 in 2024-03-21T13:00:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:04.396 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:04 smithi172 podman[102697]: 2024-03-21 13:00:04.041188324 +0000 UTC m=+0.742770146 container remove 3ea41cf71314da6d8cabff77a146eb9ce04cec489dcc4597aaeae9fbf39e9cec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, vcs-type=git, release=754, version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, ceph=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD) 2024-03-21T13:00:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:00:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[84635]: osdmap e112: 8 total, 7 up, 8 in 2024-03-21T13:00:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:04.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:00:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[90110]: osdmap e112: 8 total, 7 up, 8 in 2024-03-21T13:00:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:04.527 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:00:04.527 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:00:04.527 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 1 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 6 2024-03-21T13:00:04.528 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:00:04.529 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:00:04.741 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:04 smithi172 podman[102855]: 2024-03-21 13:00:04.391635786 +0000 UTC m=+0.024918396 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:00:04.742 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:04 smithi172 podman[102855]: 2024-03-21 13:00:04.517092334 +0000 UTC m=+0.150374944 container create 33b5e4a61e224f46ff9410c8192e43bf3dd9540ce639d4cea2c6efd6b4de4408 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:00:04.742 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:04 smithi172 podman[102855]: 2024-03-21 13:00:04.738318639 +0000 UTC m=+0.371601277 container init 33b5e4a61e224f46ff9410c8192e43bf3dd9540ce639d4cea2c6efd6b4de4408 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-21T13:00:04.742 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:04 smithi172 podman[102855]: 2024-03-21 13:00:04.74211015 +0000 UTC m=+0.375392766 container start 33b5e4a61e224f46ff9410c8192e43bf3dd9540ce639d4cea2c6efd6b4de4408 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-21T13:00:04.869 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "1/8 daemons upgraded", 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:00:04.870 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:00:05.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[84635]: pgmap v85: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:05.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[84635]: from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:05.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[84635]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:05.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[84635]: osdmap e113: 8 total, 7 up, 8 in 2024-03-21T13:00:05.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1975933715' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:05.042 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:04 smithi172 podman[102855]: 2024-03-21 13:00:04.861514114 +0000 UTC m=+0.494796729 container attach 33b5e4a61e224f46ff9410c8192e43bf3dd9540ce639d4cea2c6efd6b4de4408 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:00:05.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[90110]: pgmap v85: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:05.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[90110]: from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:05.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[90110]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:05.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[90110]: osdmap e113: 8 total, 7 up, 8 in 2024-03-21T13:00:05.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:04 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1975933715' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:05.311 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:05 smithi172 podman[102855]: 2024-03-21 13:00:05.041673473 +0000 UTC m=+0.674956082 container died 33b5e4a61e224f46ff9410c8192e43bf3dd9540ce639d4cea2c6efd6b4de4408 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T13:00:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:04 smithi177 ceph-mon[67966]: pgmap v85: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T13:00:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:04 smithi177 ceph-mon[67966]: from='client.64216 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:04 smithi177 ceph-mon[67966]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:04 smithi177 ceph-mon[67966]: osdmap e113: 8 total, 7 up, 8 in 2024-03-21T13:00:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:04 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1975933715' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:05.581 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:00:05] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-03-21T13:00:05.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:05 smithi172 podman[102908]: 2024-03-21 13:00:05.582258453 +0000 UTC m=+0.532144603 container remove 33b5e4a61e224f46ff9410c8192e43bf3dd9540ce639d4cea2c6efd6b4de4408 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T13:00:05.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:05 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-21T13:00:05.897 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:05 smithi172 systemd[1]: Stopped Ceph osd.1 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:00:05.897 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:05 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.1.service: Consumed 14.767s CPU time. 2024-03-21T13:00:05.897 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:05 smithi172 systemd[1]: Starting Ceph osd.1 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:00:06.211 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:05 smithi172 ceph-mon[84635]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:06.211 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:05 smithi172 ceph-mon[84635]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:06.211 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:05 smithi172 ceph-mon[90110]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:06.211 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:05 smithi172 ceph-mon[90110]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:06.211 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 podman[103015]: 2024-03-21 13:00:05.919413109 +0000 UTC m=+0.017800019 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:00:06.211 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 podman[103015]: 2024-03-21 13:00:06.024972402 +0000 UTC m=+0.123359344 container create 355d4beefed1b73a51f12ba164054ce8d0baf56162eb540b7ec2f644506e4e15 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319) 2024-03-21T13:00:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:05 smithi177 ceph-mon[67966]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:05 smithi177 ceph-mon[67966]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:06.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 podman[103015]: 2024-03-21 13:00:06.211182448 +0000 UTC m=+0.309569357 container init 355d4beefed1b73a51f12ba164054ce8d0baf56162eb540b7ec2f644506e4e15 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:00:06.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 podman[103015]: 2024-03-21 13:00:06.214520209 +0000 UTC m=+0.312907110 container start 355d4beefed1b73a51f12ba164054ce8d0baf56162eb540b7ec2f644506e4e15 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-21T13:00:06.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 podman[103015]: 2024-03-21 13:00:06.310623987 +0000 UTC m=+0.409010892 container attach 355d4beefed1b73a51f12ba164054ce8d0baf56162eb540b7ec2f644506e4e15 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:00:06.951 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:00:06.951 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 bash[103015]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-21T13:00:06.952 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-21T13:00:06.953 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate[103029]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-21T13:00:06.953 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:06 smithi172 podman[103015]: 2024-03-21 13:00:06.720600192 +0000 UTC m=+0.818987093 container died 355d4beefed1b73a51f12ba164054ce8d0baf56162eb540b7ec2f644506e4e15 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:00:07.310 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-mon[84635]: pgmap v87: 161 pgs: 12 active+undersized+degraded, 19 activating+undersized, 32 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:07.310 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-mon[84635]: Health check failed: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2024-03-21T13:00:07.310 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 83/627 objects degraded (13.238%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:07.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-mon[90110]: pgmap v87: 161 pgs: 12 active+undersized+degraded, 19 activating+undersized, 32 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:07.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-mon[90110]: Health check failed: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2024-03-21T13:00:07.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:06 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 83/627 objects degraded (13.238%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:06 smithi177 ceph-mon[67966]: pgmap v87: 161 pgs: 12 active+undersized+degraded, 19 activating+undersized, 32 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:06 smithi177 ceph-mon[67966]: Health check failed: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2024-03-21T13:00:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:06 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 83/627 objects degraded (13.238%), 12 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:07.629 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 podman[103165]: 2024-03-21 13:00:07.310504859 +0000 UTC m=+0.581213294 container remove 355d4beefed1b73a51f12ba164054ce8d0baf56162eb540b7ec2f644506e4e15 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:00:07.629 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 podman[103240]: 2024-03-21 13:00:07.515548254 +0000 UTC m=+0.110046404 container create 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:00:07.629 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 podman[103240]: 2024-03-21 13:00:07.426721706 +0000 UTC m=+0.021219864 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:00:07.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 podman[103240]: 2024-03-21 13:00:07.768603326 +0000 UTC m=+0.363101481 container init 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T13:00:07.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 podman[103240]: 2024-03-21 13:00:07.771918715 +0000 UTC m=+0.366416868 container start 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T13:00:07.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 ceph-osd[103263]: -- 172.21.15.172:0/1768113651 <== mon.0 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560bc632f680 con 0x560bc62f8c00 2024-03-21T13:00:07.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 bash[103240]: 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d 2024-03-21T13:00:07.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:07 smithi172 systemd[1]: Started Ceph osd.1 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:00:08.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:00:08.311+0000 7f6af4b337c0 -1 Falling back to public interface 2024-03-21T13:00:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[84635]: pgmap v88: 161 pgs: 12 active+undersized+degraded, 19 activating+undersized, 32 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[90110]: pgmap v88: 161 pgs: 12 active+undersized+degraded, 19 activating+undersized, 32 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:08 smithi177 ceph-mon[67966]: pgmap v88: 161 pgs: 12 active+undersized+degraded, 19 activating+undersized, 32 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:10.301 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:09 smithi172 ceph-mon[84635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive) 2024-03-21T13:00:10.302 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:09 smithi172 ceph-mon[90110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive) 2024-03-21T13:00:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:09 smithi177 ceph-mon[67966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive) 2024-03-21T13:00:11.256 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:10 smithi172 ceph-mon[84635]: pgmap v89: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:11.256 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:10 smithi172 ceph-mon[90110]: pgmap v89: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:11.256 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:00:11.046+0000 7f6af4b337c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-21T13:00:11.256 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:00:11.077+0000 7f6af4b337c0 -1 osd.1 111 log_to_monitors true 2024-03-21T13:00:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:10 smithi177 ceph-mon[67966]: pgmap v89: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:11 smithi172 ceph-mon[84635]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:00:12.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:11 smithi172 ceph-mon[90110]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:00:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:11 smithi177 ceph-mon[67966]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:00:12.896 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:00:12.437+0000 7f6ae1f83700 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:00:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: pgmap v90: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:00:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: osdmap e114: 8 total, 7 up, 8 in 2024-03-21T13:00:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:00:13.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:13.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:13.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' 2024-03-21T13:00:13.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:12 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: pgmap v90: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: osdmap e114: 8 total, 7 up, 8 in 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' 2024-03-21T13:00:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: pgmap v90: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 119 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: osdmap e114: 8 total, 7 up, 8 in 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: from='osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625]' entity='osd.1' 2024-03-21T13:00:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:12 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 83/627 objects degraded (13.238%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:14.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[84635]: osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625] boot 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[84635]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[90110]: osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625] boot 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[90110]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:00:14.243 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:14.244 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:13 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:00:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:13 smithi177 ceph-mon[67966]: osd.1 [v2:172.21.15.172:6810/2974776625,v1:172.21.15.172:6811/2974776625] boot 2024-03-21T13:00:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:13 smithi177 ceph-mon[67966]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T13:00:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:00:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:15.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:14 smithi172 ceph-mon[84635]: pgmap v92: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:15.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:14 smithi172 ceph-mon[84635]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:00:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:14 smithi172 ceph-mon[90110]: pgmap v92: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:14 smithi172 ceph-mon[90110]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:00:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:14 smithi177 ceph-mon[67966]: pgmap v92: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 83/627 objects degraded (13.238%) 2024-03-21T13:00:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:14 smithi177 ceph-mon[67966]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T13:00:15.597 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:00:15] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-03-21T13:00:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:16 smithi172 ceph-mon[84635]: pgmap v95: 161 pgs: 43 peering, 6 active+undersized+degraded, 14 active+undersized, 98 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-21T13:00:17.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:16 smithi172 ceph-mon[90110]: pgmap v95: 161 pgs: 43 peering, 6 active+undersized+degraded, 14 active+undersized, 98 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-21T13:00:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:16 smithi177 ceph-mon[67966]: pgmap v95: 161 pgs: 43 peering, 6 active+undersized+degraded, 14 active+undersized, 98 active+clean; 585 KiB data, 134 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/627 objects degraded (4.625%) 2024-03-21T13:00:18.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: pgmap v96: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:18.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:18.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:18.497 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: pgmap v97: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: pgmap v98: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded) 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:18.498 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: pgmap v96: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: pgmap v97: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: pgmap v98: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:18.499 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded) 2024-03-21T13:00:18.500 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.500 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: pgmap v96: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: pgmap v97: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-21T13:00:18.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: pgmap v98: 161 pgs: 43 peering, 118 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/627 objects degraded (4.625%), 6 pgs degraded) 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:18.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:19.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:00:19.187+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:19.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:19.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:19.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:19.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:19.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:19.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:19.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:00:19.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:19.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:19.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:19.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:00:19.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.860 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:19.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:19.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:00:19.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:19.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:00:19.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:00:19.901 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:19.902 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:20.622 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:20.623 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:20.624 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:20.625 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:20.626 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:20.627 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:00:20.628 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.629 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:20.630 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: Upgrade: Finalizing container_image settings 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: Upgrade: Complete! 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:20.631 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: Upgrade: Finalizing container_image settings 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: Upgrade: Complete! 2024-03-21T13:00:20.632 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:00:20.633 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:00:20.633 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:20.633 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:20 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:20.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:00:20.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: Upgrade: Finalizing container_image settings 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: Upgrade: Complete! 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:00:20.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:20.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:20 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:00:21.295+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:21.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:21.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:21.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:21.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:21.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:21 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:21.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:21.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:21.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:21.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:22.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:22.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:22.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:22.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:22.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:22.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:22.653 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:00:22.654 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:22.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:22.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:22.657 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:22.657 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:22.657 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:22 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-03-21T13:00:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:22.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:22.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:22.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:22.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:22.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:22.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:00:22.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:22.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:22.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:22.863 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:22 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:00:23.109+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:23.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:23.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:00:23.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:23.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:23.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:24 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:24 smithi177 ceph-mon[67966]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-21T13:00:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:24 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:24.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:24.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:24 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:24.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:24.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:24.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:24.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:24 smithi172 ceph-mon[90110]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:24.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:24 smithi172 ceph-mon[84635]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-03-21T13:00:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:00:25] "GET /metrics HTTP/1.1" 200 38191 "" "Prometheus/2.43.0" 2024-03-21T13:00:26.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:26 smithi172 ceph-mon[84635]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-03-21T13:00:26.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:26 smithi172 ceph-mon[90110]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-03-21T13:00:26.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:26 smithi177 ceph-mon[67966]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 0 op/s 2024-03-21T13:00:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:28 smithi172 ceph-mon[84635]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-03-21T13:00:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:28 smithi172 ceph-mon[90110]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-03-21T13:00:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:28 smithi177 ceph-mon[67966]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 634 B/s rd, 0 op/s 2024-03-21T13:00:28.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:30.625 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:30 smithi172 ceph-mon[84635]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:30.626 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:30 smithi172 ceph-mon[90110]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:30.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:30 smithi177 ceph-mon[67966]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:31.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:31.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:31.758 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:32.633 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:32 smithi172 ceph-mon[84635]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:32.633 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:32 smithi172 ceph-mon[90110]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:32.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:32 smithi177 ceph-mon[67966]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:00:33.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:33.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:34.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:34 smithi172 ceph-mon[84635]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:34.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:34 smithi172 ceph-mon[90110]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:34.857 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:34 smithi177 ceph-mon[67966]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:35.544 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:00:35] "GET /metrics HTTP/1.1" 200 38184 "" "Prometheus/2.43.0" 2024-03-21T13:00:35.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T13:00:36.600 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:36 smithi172 ceph-mon[84635]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:36.600 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:36 smithi172 ceph-mon[84635]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:36.600 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:36 smithi172 ceph-mon[90110]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:36.600 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:36 smithi172 ceph-mon[90110]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:36 smithi177 ceph-mon[67966]: from='client.64252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:36.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:36 smithi177 ceph-mon[67966]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (5m) 25s ago 27m 26.9M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (4m) 2m ago 26m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 25s ago 26m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (6m) 2m ago 31m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (16m) 25s ago 33m 505M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (3m) 25s ago 33m 47.7M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:00:38.008 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (3m) 2m ago 31m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (2m) 25s ago 32m 38.5M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (5m) 25s ago 27m 15.5M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (5m) 2m ago 27m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (52s) 25s ago 31m 50.0M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (30s) 25s ago 30m 11.6M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (30m) 25s ago 30m 50.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (30m) 25s ago 30m 54.7M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (29m) 2m ago 29m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (29m) 2m ago 29m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (29m) 2m ago 29m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (28m) 2m ago 28m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (5m) 2m ago 27m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (26m) 25s ago 26m 77.5M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:00:38.009 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (26m) 2m ago 26m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:00:38.644 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:38 smithi172 ceph-mon[84635]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:38.644 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:38 smithi172 ceph-mon[84635]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:38.644 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:38 smithi172 ceph-mon[90110]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:38.644 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:38 smithi172 ceph-mon[90110]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:38.669 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-21T13:00:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:38 smithi177 ceph-mon[67966]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:38.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:38 smithi177 ceph-mon[67966]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:40.636 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:40 smithi172 ceph-mon[84635]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:40.637 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:40 smithi172 ceph-mon[90110]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:40.645 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:00:40.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:40 smithi177 ceph-mon[67966]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:41.359 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-03-21T13:00:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:41 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3182364629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:41.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:41 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:41 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3182364629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:41.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:41 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:41 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3182364629' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:00:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:41 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:42.575 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:42 smithi172 ceph-mon[84635]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:42.575 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:42 smithi172 ceph-mon[90110]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:42 smithi177 ceph-mon[67966]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:44.471 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:44 smithi172 ceph-mon[84635]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:44.471 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:44 smithi172 ceph-mon[84635]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:44.472 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:44 smithi172 ceph-mon[90110]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:44.472 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:44 smithi172 ceph-mon[90110]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:44 smithi177 ceph-mon[67966]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:44.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:44 smithi177 ceph-mon[67966]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:45.397 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:00:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:00:45] "GET /metrics HTTP/1.1" 200 38184 "" "Prometheus/2.43.0" 2024-03-21T13:00:45.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T13:00:46.594 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:46 smithi172 ceph-mon[84635]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:46.594 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:46 smithi172 ceph-mon[90110]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:46.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:46 smithi177 ceph-mon[67966]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:48.072 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:00:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T13:00:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T13:00:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T13:00:48.072 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:00:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T13:00:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T13:00:48.073 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:00:48.073 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:00:48.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:48 smithi172 ceph-mon[84635]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:48.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:48.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:48 smithi172 ceph-mon[90110]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:48.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:48.784 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T13:00:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:48 smithi177 ceph-mon[67966]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:00:49.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:49 smithi172 ceph-mon[84635]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:49.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:49 smithi172 ceph-mon[90110]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:49.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:49 smithi177 ceph-mon[67966]: from='client.54342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:50.513 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:50 smithi172 ceph-mon[84635]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:50.513 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:50 smithi172 ceph-mon[90110]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:50.815 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:00:50.816 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T13:00:50.816 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T13:00:50.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:50 smithi177 ceph-mon[67966]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:51.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:51 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2051521916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:00:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:51.649 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:51 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2051521916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:00:51.692 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-03-21T13:00:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:00:51.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:51 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2051521916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:00:52.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:52 smithi172 ceph-mon[90110]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:52.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:52 smithi172 ceph-mon[84635]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:52.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:52 smithi177 ceph-mon[67966]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:54.682 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:54 smithi172 ceph-mon[84635]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:54.682 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:54 smithi172 ceph-mon[84635]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:54.682 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:54 smithi172 ceph-mon[90110]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:54.682 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:54 smithi172 ceph-mon[90110]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:54 smithi177 ceph-mon[67966]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:00:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:54 smithi177 ceph-mon[67966]: from='client.64288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:00:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:00:55] "GET /metrics HTTP/1.1" 200 38186 "" "Prometheus/2.43.0" 2024-03-21T13:00:55.791 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:00:56.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:57.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:57.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:57.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:57.111 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:00:57.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:00:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:57 smithi177 ceph-mon[67966]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-21T13:00:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:57 smithi177 ceph-mon[67966]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:57 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:58.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:57 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:58.128 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[90110]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[90110]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[84635]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 922 B/s rd, 0 op/s 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[84635]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:00:58.129 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:57 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:00:58.577 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:00:59.238 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (5m) 47s ago 27m 26.9M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (5m) 3m ago 27m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 47s ago 26m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (6m) 3m ago 31m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (16m) 47s ago 33m 505M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (3m) 47s ago 33m 47.7M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (4m) 3m ago 32m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (2m) 47s ago 32m 38.5M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (6m) 47s ago 28m 15.5M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (6m) 3m ago 28m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (73s) 47s ago 31m 50.0M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (51s) 47s ago 31m 11.6M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (30m) 47s ago 30m 50.9M 2992M 17.2.0 e1d6a67b021e caf5c9d5936c 2024-03-21T13:00:59.239 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (30m) 47s ago 30m 54.7M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (30m) 3m ago 30m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (29m) 3m ago 29m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (29m) 3m ago 29m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (28m) 3m ago 28m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (5m) 3m ago 28m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (26m) 47s ago 26m 77.5M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:00:59.240 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (26m) 3m ago 26m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:00:59.693 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:00:59.693 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:00:59.693 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:00:59.694 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:00:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-21T13:00:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 7 2024-03-21T13:00:59.695 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:00:59.695 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:00:59.695 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:00:59.554+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:59.696 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:00:59.697 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:00:59.698 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:00:59.699 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:00:59.700 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:00:59.700 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:00:59.700 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:00:59.700 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:00.053 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:01:00.053 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", 2024-03-21T13:01:00.053 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:01:00.054 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-03-21T13:01:00.054 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:01:00.054 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "", 2024-03-21T13:01:00.054 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 image", 2024-03-21T13:01:00.054 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:01:00.054 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:01:00.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[84635]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T13:01:00.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[84635]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:00.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[84635]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:00.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/607085148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:00.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[90110]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T13:01:00.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[90110]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:00.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[90110]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:00.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:00:59 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/607085148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:59 smithi177 ceph-mon[67966]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 562 B/s rd, 0 op/s 2024-03-21T13:01:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:59 smithi177 ceph-mon[67966]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:59 smithi177 ceph-mon[67966]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:00.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:00:59 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/607085148' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:00 smithi177 ceph-mon[67966]: from='client.64306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:00 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:01.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:01.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:01.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:01.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:01.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:00 smithi177 ceph-mon[67966]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:00 smithi172 ceph-mon[84635]: from='client.64306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:01.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:00 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:01.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:01.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:01.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:01.151 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:00 smithi172 ceph-mon[84635]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:00 smithi172 ceph-mon[90110]: from='client.64306 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:00 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:01.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:01.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:01.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:00 smithi172 ceph-mon[90110]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:01.946 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:01.946 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.947 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:01.948 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:01 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T13:01:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:02.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:01 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T13:01:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:01:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:01:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mgr 2024-03-21T13:01:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mon 2024-03-21T13:01:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T13:01:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T13:01:03.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:02 smithi177 ceph-mon[67966]: Upgrade: osd.2 is safe to restart 2024-03-21T13:01:03.124 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:01:03.124 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mgr 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mon 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all crash 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[84635]: Upgrade: osd.2 is safe to restart 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mgr 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mon 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all crash 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-21T13:01:03.125 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:02 smithi172 ceph-mon[90110]: Upgrade: osd.2 is safe to restart 2024-03-21T13:01:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:03 smithi177 ceph-mon[67966]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T13:01:04.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[84635]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T13:01:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[90110]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T13:01:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:04 smithi177 ceph-mon[67966]: Upgrade: Updating osd.2 2024-03-21T13:01:05.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:04 smithi177 ceph-mon[67966]: Deploying daemon osd.2 on smithi172 2024-03-21T13:01:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:04 smithi172 ceph-mon[84635]: Upgrade: Updating osd.2 2024-03-21T13:01:05.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:04 smithi172 ceph-mon[84635]: Deploying daemon osd.2 on smithi172 2024-03-21T13:01:05.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:04 smithi172 ceph-mon[90110]: Upgrade: Updating osd.2 2024-03-21T13:01:05.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:04 smithi172 ceph-mon[90110]: Deploying daemon osd.2 on smithi172 2024-03-21T13:01:05.542 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:01:05] "GET /metrics HTTP/1.1" 200 38187 "" "Prometheus/2.43.0" 2024-03-21T13:01:06.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:05 smithi177 ceph-mon[67966]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:05 smithi172 ceph-mon[84635]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:05 smithi172 ceph-mon[90110]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:07.146 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:06 smithi172 systemd[1]: Stopping Ceph osd.2 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:01:07.146 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[37964]: 2024-03-21T13:01:06.908+0000 7f776b686700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:01:07.146 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[37964]: 2024-03-21T13:01:06.908+0000 7f776b686700 -1 osd.2 116 *** Got signal Terminated *** 2024-03-21T13:01:07.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:06 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[37964]: 2024-03-21T13:01:06.908+0000 7f776b686700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:01:08.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:07 smithi177 ceph-mon[67966]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:08.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:07 smithi177 ceph-mon[67966]: osd.2 marked itself down and dead 2024-03-21T13:01:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:07 smithi172 ceph-mon[84635]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:07 smithi172 ceph-mon[84635]: osd.2 marked itself down and dead 2024-03-21T13:01:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:07 smithi172 ceph-mon[90110]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:08.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:07 smithi172 ceph-mon[90110]: osd.2 marked itself down and dead 2024-03-21T13:01:08.471 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:08 smithi172 podman[110116]: 2024-03-21 13:01:08.215522998 +0000 UTC m=+1.407499273 container died caf5c9d5936c72e8265bcbf943074c97fc73ced66765352d0ca62c78c2e2dd07 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, version=8, io.buildah.version=1.19.8, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-21T13:01:08.828 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:08 smithi172 podman[110116]: 2024-03-21 13:01:08.56712792 +0000 UTC m=+1.759104193 container cleanup caf5c9d5936c72e8265bcbf943074c97fc73ced66765352d0ca62c78c2e2dd07 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, version=8, release=754, distribution-scope=public, ceph=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-21T13:01:08.828 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:08 smithi172 bash[110116]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2 2024-03-21T13:01:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:08 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:01:09.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:08 smithi177 ceph-mon[67966]: osdmap e117: 8 total, 7 up, 8 in 2024-03-21T13:01:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:08 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:01:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:08 smithi172 ceph-mon[84635]: osdmap e117: 8 total, 7 up, 8 in 2024-03-21T13:01:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:08 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:01:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:08 smithi172 ceph-mon[90110]: osdmap e117: 8 total, 7 up, 8 in 2024-03-21T13:01:09.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:08 smithi172 podman[110133]: 2024-03-21 13:01:08.846429535 +0000 UTC m=+0.632552750 container remove caf5c9d5936c72e8265bcbf943074c97fc73ced66765352d0ca62c78c2e2dd07 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-type=git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_CLEAN=True, release=754, io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-03-21T13:01:09.455 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:09 smithi172 podman[110224]: 2024-03-21 13:01:09.085732958 +0000 UTC m=+0.017002044 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:01:09.455 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:09 smithi172 podman[110224]: 2024-03-21 13:01:09.185979829 +0000 UTC m=+0.117248895 container create 914eb21e3ce5059e5704b299ed7bc574281ba4d063e0e6c41669e7b2a1e4390a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:01:09.757 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:09 smithi172 podman[110224]: 2024-03-21 13:01:09.455515781 +0000 UTC m=+0.386784862 container init 914eb21e3ce5059e5704b299ed7bc574281ba4d063e0e6c41669e7b2a1e4390a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T13:01:09.757 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:09 smithi172 podman[110224]: 2024-03-21 13:01:09.459126097 +0000 UTC m=+0.390395172 container start 914eb21e3ce5059e5704b299ed7bc574281ba4d063e0e6c41669e7b2a1e4390a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, ceph=True) 2024-03-21T13:01:09.758 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:09 smithi172 podman[110224]: 2024-03-21 13:01:09.530203829 +0000 UTC m=+0.461472897 container attach 914eb21e3ce5059e5704b299ed7bc574281ba4d063e0e6c41669e7b2a1e4390a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-21T13:01:10.050 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:09 smithi172 ceph-mon[84635]: pgmap v129: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:10.050 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:09 smithi172 ceph-mon[84635]: osdmap e118: 8 total, 7 up, 8 in 2024-03-21T13:01:10.050 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:09 smithi172 ceph-mon[90110]: pgmap v129: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:10.050 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:09 smithi172 ceph-mon[90110]: osdmap e118: 8 total, 7 up, 8 in 2024-03-21T13:01:10.050 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:09 smithi172 podman[110224]: 2024-03-21 13:01:09.758123422 +0000 UTC m=+0.689392495 container died 914eb21e3ce5059e5704b299ed7bc574281ba4d063e0e6c41669e7b2a1e4390a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-21T13:01:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:09 smithi177 ceph-mon[67966]: pgmap v129: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:10.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:09 smithi177 ceph-mon[67966]: osdmap e118: 8 total, 7 up, 8 in 2024-03-21T13:01:10.302 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 podman[110268]: 2024-03-21 13:01:10.26696596 +0000 UTC m=+0.503478482 container remove 914eb21e3ce5059e5704b299ed7bc574281ba4d063e0e6c41669e7b2a1e4390a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:01:10.302 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-21T13:01:10.302 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 systemd[1]: Stopped Ceph osd.2 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:01:10.302 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.2.service: Consumed 12.274s CPU time. 2024-03-21T13:01:10.646 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 systemd[1]: Starting Ceph osd.2 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:01:10.954 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:10 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 14 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:10.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:10 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 14 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:10.954 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 podman[110372]: 2024-03-21 13:01:10.709942221 +0000 UTC m=+0.114922170 container create e6d06088ac7cf7fc1006f3a7cfc887889bedd5a93b352221763ee90fad9f1bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-21T13:01:10.954 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 podman[110372]: 2024-03-21 13:01:10.614002932 +0000 UTC m=+0.018982889 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:01:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:10 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 39/627 objects degraded (6.220%), 14 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:11.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 podman[110372]: 2024-03-21 13:01:10.954454701 +0000 UTC m=+0.359434659 container init e6d06088ac7cf7fc1006f3a7cfc887889bedd5a93b352221763ee90fad9f1bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-21T13:01:11.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:10 smithi172 podman[110372]: 2024-03-21 13:01:10.957700537 +0000 UTC m=+0.362680487 container start e6d06088ac7cf7fc1006f3a7cfc887889bedd5a93b352221763ee90fad9f1bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:01:11.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 podman[110372]: 2024-03-21 13:01:11.046047299 +0000 UTC m=+0.451027270 container attach e6d06088ac7cf7fc1006f3a7cfc887889bedd5a93b352221763ee90fad9f1bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-21T13:01:11.741 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:01:11.741 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 bash[110372]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T13:01:11.742 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T13:01:11.743 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-21T13:01:11.743 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T13:01:11.743 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate[110387]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-21T13:01:11.743 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:11 smithi172 podman[110372]: 2024-03-21 13:01:11.47658294 +0000 UTC m=+0.881562903 container died e6d06088ac7cf7fc1006f3a7cfc887889bedd5a93b352221763ee90fad9f1bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T13:01:12.047 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-mon[84635]: pgmap v131: 161 pgs: 14 active+undersized+degraded, 2 stale+active+clean, 27 active+undersized, 118 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 39/627 objects degraded (6.220%) 2024-03-21T13:01:12.047 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:12.047 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-mon[90110]: pgmap v131: 161 pgs: 14 active+undersized+degraded, 2 stale+active+clean, 27 active+undersized, 118 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 39/627 objects degraded (6.220%) 2024-03-21T13:01:12.048 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:11 smithi177 ceph-mon[67966]: pgmap v131: 161 pgs: 14 active+undersized+degraded, 2 stale+active+clean, 27 active+undersized, 118 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 39/627 objects degraded (6.220%) 2024-03-21T13:01:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:12.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 podman[110511]: 2024-03-21 13:01:12.049526038 +0000 UTC m=+0.568188554 container remove e6d06088ac7cf7fc1006f3a7cfc887889bedd5a93b352221763ee90fad9f1bc5 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-21T13:01:12.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 podman[110576]: 2024-03-21 13:01:12.165550868 +0000 UTC m=+0.018070799 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:01:12.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 podman[110576]: 2024-03-21 13:01:12.283744534 +0000 UTC m=+0.136264464 container create 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T13:01:12.897 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 podman[110576]: 2024-03-21 13:01:12.611586814 +0000 UTC m=+0.464106741 container init 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-21T13:01:12.897 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 podman[110576]: 2024-03-21 13:01:12.614738625 +0000 UTC m=+0.467258549 container start 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-21T13:01:12.897 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 ceph-osd[110595]: -- 172.21.15.172:0/1668998709 <== mon.1 v2:172.21.15.172:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562440a89680 con 0x562440a52c00 2024-03-21T13:01:12.897 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 bash[110576]: 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 2024-03-21T13:01:12.897 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:12 smithi172 systemd[1]: Started Ceph osd.2 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:01:13.591 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:01:13.165+0000 7f2484bd07c0 -1 Falling back to public interface 2024-03-21T13:01:13.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[84635]: pgmap v132: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[90110]: pgmap v132: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:13.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:13 smithi177 ceph-mon[67966]: pgmap v132: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:14.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:01:15] "GET /metrics HTTP/1.1" 200 38187 "" "Prometheus/2.43.0" 2024-03-21T13:01:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:15 smithi177 ceph-mon[67966]: pgmap v133: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:16.109 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:15 smithi172 ceph-mon[84635]: pgmap v133: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:16.109 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:15 smithi172 ceph-mon[90110]: pgmap v133: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:16.110 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:01:15.870+0000 7f2484bd07c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-21T13:01:16.110 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:01:15.891+0000 7f2484bd07c0 -1 osd.2 116 log_to_monitors true 2024-03-21T13:01:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:16 smithi177 ceph-mon[67966]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T13:01:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:16 smithi172 ceph-mon[84635]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T13:01:17.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:16 smithi172 ceph-mon[90110]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T13:01:17.147 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:01:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:01:16.799+0000 7f2476028700 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:01:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:17 smithi177 ceph-mon[67966]: pgmap v134: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:17 smithi177 ceph-mon[67966]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T13:01:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:17 smithi177 ceph-mon[67966]: osdmap e119: 8 total, 7 up, 8 in 2024-03-21T13:01:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:17 smithi177 ceph-mon[67966]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:01:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:17 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:17 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[90110]: pgmap v134: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[90110]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T13:01:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[90110]: osdmap e119: 8 total, 7 up, 8 in 2024-03-21T13:01:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[90110]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:01:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[84635]: pgmap v134: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[84635]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T13:01:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[84635]: osdmap e119: 8 total, 7 up, 8 in 2024-03-21T13:01:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[84635]: from='osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:01:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:17 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013] boot 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 58/627 objects degraded (9.250%), 16 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[84635]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:01:19.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013] boot 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 58/627 objects degraded (9.250%), 16 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:18 smithi172 ceph-mon[90110]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: osd.2 [v2:172.21.15.172:6818/3176156013,v1:172.21.15.172:6819/3176156013] boot 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 58/627 objects degraded (9.250%), 16 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:18 smithi177 ceph-mon[67966]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T13:01:20.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:19 smithi177 ceph-mon[67966]: pgmap v137: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:20.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:19 smithi172 ceph-mon[84635]: pgmap v137: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:20.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:19 smithi172 ceph-mon[90110]: pgmap v137: 161 pgs: 16 active+undersized+degraded, 31 active+undersized, 114 active+clean; 585 KiB data, 155 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 58/627 objects degraded (9.250%) 2024-03-21T13:01:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:21 smithi177 ceph-mon[67966]: pgmap v139: 161 pgs: 3 active+undersized+degraded, 13 active+undersized, 145 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 20/627 objects degraded (3.190%) 2024-03-21T13:01:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:21 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:21 smithi172 ceph-mon[84635]: pgmap v139: 161 pgs: 3 active+undersized+degraded, 13 active+undersized, 145 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 20/627 objects degraded (3.190%) 2024-03-21T13:01:22.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:21 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:21 smithi172 ceph-mon[90110]: pgmap v139: 161 pgs: 3 active+undersized+degraded, 13 active+undersized, 145 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 20/627 objects degraded (3.190%) 2024-03-21T13:01:22.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:21 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: pgmap v140: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 19/627 objects degraded (3.030%) 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: pgmap v141: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s; 19/627 objects degraded (3.030%) 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:22.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 20/627 objects degraded (3.190%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: pgmap v140: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 19/627 objects degraded (3.030%) 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: pgmap v141: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s; 19/627 objects degraded (3.030%) 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:22.971 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:22 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 20/627 objects degraded (3.190%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: pgmap v140: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 19/627 objects degraded (3.030%) 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: pgmap v141: 161 pgs: 2 active+undersized+degraded, 2 active+undersized, 157 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 0 op/s; 19/627 objects degraded (3.030%) 2024-03-21T13:01:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:23.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:23.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:23.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:22 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 20/627 objects degraded (3.190%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T13:01:24.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:01:23.885+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:24.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:01:24.149 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:24.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:01:24.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:24.154 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:01:24.155 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:24.156 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:24.157 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:01:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:25.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.312 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.313 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:25.314 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.315 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.316 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.317 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:25.318 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.319 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.320 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: Upgrade: Finalizing container_image settings 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: Upgrade: Complete! 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 0 op/s 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/627 objects degraded (3.030%), 2 pgs degraded) 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.321 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: Upgrade: Finalizing container_image settings 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: Upgrade: Complete! 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 0 op/s 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:25.322 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:24 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/627 objects degraded (3.030%), 2 pgs degraded) 2024-03-21T13:01:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:25.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:25.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: Upgrade: Finalizing container_image settings 2024-03-21T13:01:25.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: Upgrade: Complete! 2024-03-21T13:01:25.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 0 op/s 2024-03-21T13:01:25.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 927 B/s rd, 0 op/s 2024-03-21T13:01:25.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:25.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:25.364 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:24 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 19/627 objects degraded (3.030%), 2 pgs degraded) 2024-03-21T13:01:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:01:25] "GET /metrics HTTP/1.1" 200 38058 "" "Prometheus/2.43.0" 2024-03-21T13:01:25.986 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:01:25.794+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.986 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:25.986 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.986 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.987 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.988 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.988 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.988 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.988 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.988 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:25.989 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:25.990 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:25.991 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:26.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:26.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:26.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:26.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:25 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:25 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:27.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:27.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:27.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[84635]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[84635]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:27.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[90110]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[90110]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:27.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:26 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:26 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:27.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:26 smithi177 ceph-mon[67966]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T13:01:27.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:26 smithi177 ceph-mon[67966]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-21T13:01:27.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:26 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:27.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:26 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:27.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:01:27.707+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:27.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:27.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:27.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:27.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:27.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:27.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:27.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:27.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:27.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:27.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:27 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:28 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:29.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:29.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:29.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:29.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:28 smithi177 ceph-mon[67966]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-03-21T13:01:29.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:28 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:29.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:29.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:29.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:29.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:28 smithi172 ceph-mon[84635]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:28 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:29.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:29.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:28 smithi172 ceph-mon[90110]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-03-21T13:01:29.405 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:31.158 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T13:01:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:30 smithi177 ceph-mon[67966]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:01:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:30 smithi172 ceph-mon[84635]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:01:31.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:30 smithi172 ceph-mon[90110]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:01:32.136 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:31 smithi172 ceph-mon[84635]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:32.136 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:32.137 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:31 smithi172 ceph-mon[90110]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:32.137 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:32 smithi177 ceph-mon[67966]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:32 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:33.269 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:32 smithi172 ceph-mon[84635]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:01:33.269 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:32 smithi172 ceph-mon[90110]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:01:33.269 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:01:33.269 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (6m) 16s ago 28m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:01:33.269 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (5m) 3m ago 27m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 16s ago 27m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (7m) 3m ago 32m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (17m) 16s ago 34m 508M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (3m) 16s ago 34m 50.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (4m) 3m ago 32m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (3m) 16s ago 32m 40.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (6m) 16s ago 28m 15.7M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (6m) 3m ago 28m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (107s) 16s ago 32m 51.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (85s) 16s ago 31m 50.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (21s) 16s ago 31m 14.4M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (31m) 16s ago 31m 55.0M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (30m) 3m ago 30m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T13:01:33.270 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (30m) 3m ago 30m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:01:33.271 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (29m) 3m ago 29m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:01:33.271 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (29m) 3m ago 29m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:01:33.271 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (6m) 3m ago 28m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:01:33.271 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (27m) 16s ago 27m 77.8M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:01:33.271 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (27m) 3m ago 27m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:01:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:33 smithi177 ceph-mon[67966]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:01:33.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-21T13:01:34.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:34.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:34.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:34.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:34.290 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:34.290 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:35.237 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:35 smithi172 ceph-mon[84635]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:35.237 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:35 smithi172 ceph-mon[84635]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:01:35.237 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:35 smithi172 ceph-mon[90110]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:35.237 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:35 smithi172 ceph-mon[90110]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:01:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:35 smithi177 ceph-mon[67966]: from='client.64336 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:35 smithi177 ceph-mon[67966]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:01:35.514 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:01:35] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-21T13:01:36.026 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:01:36.326 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:36 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1366489012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:36.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:36 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1366489012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:36 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1366489012' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:36.627 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-03-21T13:01:37.223 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:37 smithi172 ceph-mon[84635]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:37.224 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:37 smithi172 ceph-mon[90110]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:37 smithi177 ceph-mon[67966]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:01:38.183 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:38 smithi172 ceph-mon[84635]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:38.183 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:38 smithi172 ceph-mon[90110]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:38 smithi177 ceph-mon[67966]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:39 smithi177 ceph-mon[67966]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:39 smithi172 ceph-mon[84635]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:39 smithi172 ceph-mon[90110]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:40 smithi177 ceph-mon[67966]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:40.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:40 smithi172 ceph-mon[84635]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:40.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:40 smithi172 ceph-mon[90110]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:40.940 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:01:41.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:41 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:41 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:41 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:41.755 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T13:01:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:42 smithi177 ceph-mon[67966]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:42 smithi172 ceph-mon[84635]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:42.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:42 smithi172 ceph-mon[90110]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:43.594 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:01:43.594 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:01:43.595 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:01:44.216 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T13:01:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:44 smithi172 ceph-mon[84635]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:45.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:44 smithi172 ceph-mon[84635]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:45.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:44 smithi172 ceph-mon[90110]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:45.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:44 smithi172 ceph-mon[90110]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:44 smithi177 ceph-mon[67966]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:44 smithi177 ceph-mon[67966]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:45.644 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:01:45] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-21T13:01:46.322 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:01:46.322 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T13:01:46.323 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T13:01:47.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:46 smithi172 ceph-mon[84635]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:47.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:46 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2834793357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:01:47.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:46 smithi172 ceph-mon[90110]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:47.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:46 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2834793357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:01:47.098 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-03-21T13:01:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:46 smithi177 ceph-mon[67966]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:46 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2834793357' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:01:48.936 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:48 smithi172 ceph-mon[84635]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:48.937 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:48.937 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:48 smithi172 ceph-mon[90110]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:48.937 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:48 smithi177 ceph-mon[67966]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:01:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:01:51.093 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:50 smithi172 ceph-mon[84635]: from='client.54432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:51.093 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:50 smithi172 ceph-mon[84635]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:51.093 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:50 smithi172 ceph-mon[90110]: from='client.54432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:51.093 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:50 smithi172 ceph-mon[90110]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:51.152 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:50 smithi177 ceph-mon[67966]: from='client.54432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:50 smithi177 ceph-mon[67966]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:01:51.802 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:52.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:52.057 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:01:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[84635]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[84635]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-21T13:01:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[84635]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[90110]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[90110]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-21T13:01:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[90110]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:52 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:52 smithi177 ceph-mon[67966]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:52 smithi177 ceph-mon[67966]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-21T13:01:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:52 smithi177 ceph-mon[67966]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:52 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:01:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:52 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:01:53.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:01:53.637+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:53.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:53.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:53.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:01:53.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:53.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.736 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:01:54.961 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:54 smithi172 ceph-mon[90110]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:54 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:54.962 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:54.963 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:54.964 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:54.965 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:54 smithi172 ceph-mon[84635]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:54 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:54.966 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:54.967 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.968 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:54.969 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:54 smithi172 ceph-mon[90110]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:54.970 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:54 smithi172 ceph-mon[84635]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:54 smithi177 ceph-mon[67966]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:54 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:55.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:01:55.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:01:55.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:01:55.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:54 smithi177 ceph-mon[67966]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (6m) 38s ago 28m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (6m) 4m ago 28m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 38s ago 27m 68.5M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (7m) 4m ago 32m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (17m) 38s ago 34m 508M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (4m) 38s ago 34m 50.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (5m) 4m ago 33m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (3m) 38s ago 33m 40.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (7m) 38s ago 29m 15.7M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (7m) 4m ago 29m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:01:55.435 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (2m) 38s ago 32m 51.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (108s) 38s ago 32m 50.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (43s) 38s ago 31m 14.4M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (31m) 38s ago 31m 55.0M 2992M 17.2.0 e1d6a67b021e 5c38cc553f5a 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (31m) 4m ago 31m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (30m) 4m ago 30m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (30m) 4m ago 30m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (29m) 4m ago 29m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (6m) 4m ago 29m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (27m) 38s ago 27m 77.8M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:01:55.436 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (27m) 4m ago 27m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:01:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:01:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:01:55] "GET /metrics HTTP/1.1" 200 38195 "" "Prometheus/2.43.0" 2024-03-21T13:01:55.848 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:01:55.849 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:01:55.850 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-21T13:01:55.850 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 8 2024-03-21T13:01:55.850 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:01:55.850 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:01:55.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:55 smithi172 ceph-mon[90110]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:55.916 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:55 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2438008048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:55.917 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:55 smithi172 ceph-mon[84635]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:55.917 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:55 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2438008048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "3/8 daemons upgraded", 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:01:56.242 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:01:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:55 smithi177 ceph-mon[67966]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:55 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2438008048' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:01:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mgr 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mon 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T13:01:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:56 smithi177 ceph-mon[67966]: Upgrade: osd.3 is safe to restart 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mgr 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mon 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all crash 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[84635]: Upgrade: osd.3 is safe to restart 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:01:57.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mgr 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mon 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all crash 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-21T13:01:57.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:56 smithi172 ceph-mon[90110]: Upgrade: osd.3 is safe to restart 2024-03-21T13:01:58.342 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[84635]: from='client.64396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:58.342 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:58.342 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T13:01:58.343 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:58.343 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[90110]: from='client.64396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:58.343 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:58.343 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T13:01:58.343 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:58.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:57 smithi177 ceph-mon[67966]: from='client.64396 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:01:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:01:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T13:01:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:01:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:58 smithi177 ceph-mon[67966]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:58 smithi177 ceph-mon[67966]: Upgrade: Updating osd.3 2024-03-21T13:01:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:01:58 smithi177 ceph-mon[67966]: Deploying daemon osd.3 on smithi172 2024-03-21T13:01:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:58 smithi172 ceph-mon[84635]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:58 smithi172 ceph-mon[84635]: Upgrade: Updating osd.3 2024-03-21T13:01:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:01:58 smithi172 ceph-mon[84635]: Deploying daemon osd.3 on smithi172 2024-03-21T13:01:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:58 smithi172 ceph-mon[90110]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:01:59.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:58 smithi172 ceph-mon[90110]: Upgrade: Updating osd.3 2024-03-21T13:01:59.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:01:58 smithi172 ceph-mon[90110]: Deploying daemon osd.3 on smithi172 2024-03-21T13:02:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:00 smithi177 ceph-mon[67966]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:02:01.377 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:00 smithi172 ceph-mon[84635]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:02:01.377 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:00 smithi172 ceph-mon[90110]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:02:01.377 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:01 smithi172 systemd[1]: Stopping Ceph osd.3 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:02:01.646 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[40948]: 2024-03-21T13:02:01.376+0000 7f1365b98700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:02:01.646 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[40948]: 2024-03-21T13:02:01.376+0000 7f1365b98700 -1 osd.3 121 *** Got signal Terminated *** 2024-03-21T13:02:01.646 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[40948]: 2024-03-21T13:02:01.376+0000 7f1365b98700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:02:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:01 smithi177 ceph-mon[67966]: osd.3 marked itself down and dead 2024-03-21T13:02:02.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:02.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-mon[84635]: osd.3 marked itself down and dead 2024-03-21T13:02:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:01 smithi172 ceph-mon[90110]: osd.3 marked itself down and dead 2024-03-21T13:02:02.728 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:02 smithi172 podman[117396]: 2024-03-21 13:02:02.471410425 +0000 UTC m=+1.224694141 container died 5c38cc553f5a81d8402ab479cf3a700de38c69354ccfcdf78b78e1d239c18094 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, version=8, name=centos-stream, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, RELEASE=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754) 2024-03-21T13:02:02.988 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:02 smithi172 podman[117396]: 2024-03-21 13:02:02.952659526 +0000 UTC m=+1.705943246 container cleanup 5c38cc553f5a81d8402ab479cf3a700de38c69354ccfcdf78b78e1d239c18094 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_CLEAN=True, vcs-type=git, maintainer=Guillaume Abrioux , RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870) 2024-03-21T13:02:02.988 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:02 smithi172 bash[117396]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3 2024-03-21T13:02:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:02 smithi177 ceph-mon[67966]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:02:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:02 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:02:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:02 smithi177 ceph-mon[67966]: osdmap e122: 8 total, 7 up, 8 in 2024-03-21T13:02:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:02 smithi172 ceph-mon[90110]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:02:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:02 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:02:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:02 smithi172 ceph-mon[90110]: osdmap e122: 8 total, 7 up, 8 in 2024-03-21T13:02:03.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:02 smithi172 ceph-mon[84635]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:02:03.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:02 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:02:03.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:02 smithi172 ceph-mon[84635]: osdmap e122: 8 total, 7 up, 8 in 2024-03-21T13:02:03.397 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:03 smithi172 podman[117413]: 2024-03-21 13:02:03.212551113 +0000 UTC m=+0.735088807 container remove 5c38cc553f5a81d8402ab479cf3a700de38c69354ccfcdf78b78e1d239c18094 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., architecture=x86_64, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git) 2024-03-21T13:02:03.840 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:03 smithi172 podman[117505]: 2024-03-21 13:02:03.530093224 +0000 UTC m=+0.111384107 container create 36b3dd6f08de384a3eb7067291564730b26a2cef1ec0d21bb3e7a4e38fdae200 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-21T13:02:03.840 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:03 smithi172 podman[117505]: 2024-03-21 13:02:03.43818314 +0000 UTC m=+0.019474036 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:02:03.841 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:03 smithi172 podman[117505]: 2024-03-21 13:02:03.757979606 +0000 UTC m=+0.339270505 container init 36b3dd6f08de384a3eb7067291564730b26a2cef1ec0d21bb3e7a4e38fdae200 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:02:03.841 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:03 smithi172 podman[117505]: 2024-03-21 13:02:03.761498525 +0000 UTC m=+0.342789416 container start 36b3dd6f08de384a3eb7067291564730b26a2cef1ec0d21bb3e7a4e38fdae200 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:02:03.841 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:03 smithi172 podman[117505]: 2024-03-21 13:02:03.840718514 +0000 UTC m=+0.422009411 container attach 36b3dd6f08de384a3eb7067291564730b26a2cef1ec0d21bb3e7a4e38fdae200 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319) 2024-03-21T13:02:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:03 smithi172 ceph-mon[84635]: osdmap e123: 8 total, 7 up, 8 in 2024-03-21T13:02:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[90110]: osdmap e123: 8 total, 7 up, 8 in 2024-03-21T13:02:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:04.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:04 smithi172 podman[117505]: 2024-03-21 13:02:04.055907218 +0000 UTC m=+0.637198094 container died 36b3dd6f08de384a3eb7067291564730b26a2cef1ec0d21bb3e7a4e38fdae200 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=) 2024-03-21T13:02:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:03 smithi177 ceph-mon[67966]: osdmap e123: 8 total, 7 up, 8 in 2024-03-21T13:02:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:04.999 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:04 smithi172 podman[117548]: 2024-03-21 13:02:04.6945465 +0000 UTC m=+0.633532452 container remove 36b3dd6f08de384a3eb7067291564730b26a2cef1ec0d21bb3e7a4e38fdae200 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:02:04.999 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:04 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-21T13:02:04.999 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:04 smithi172 systemd[1]: Stopped Ceph osd.3 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:02:04.999 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:04 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.3.service: Consumed 16.760s CPU time. 2024-03-21T13:02:04.999 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:04 smithi172 systemd[1]: Starting Ceph osd.3 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:02:05.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[84635]: pgmap v167: 161 pgs: 13 peering, 22 stale+active+clean, 126 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:02:05.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[84635]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T13:02:05.278 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[90110]: pgmap v167: 161 pgs: 13 peering, 22 stale+active+clean, 126 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:02:05.278 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:04 smithi172 ceph-mon[90110]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T13:02:05.279 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 podman[117653]: 2024-03-21 13:02:05.040203555 +0000 UTC m=+0.018671434 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:02:05.279 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 podman[117653]: 2024-03-21 13:02:05.14540687 +0000 UTC m=+0.123874747 container create 3db686fdc65a4dd2459563d4e4b48ede1502406f34dfe56edb8cf5b756696e94 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:02:05.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:04 smithi177 ceph-mon[67966]: pgmap v167: 161 pgs: 13 peering, 22 stale+active+clean, 126 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:02:05.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:04 smithi177 ceph-mon[67966]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-21T13:02:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:02:05] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-21T13:02:05.651 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 podman[117653]: 2024-03-21 13:02:05.389720149 +0000 UTC m=+0.368188037 container init 3db686fdc65a4dd2459563d4e4b48ede1502406f34dfe56edb8cf5b756696e94 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-21T13:02:05.651 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 podman[117653]: 2024-03-21 13:02:05.39286501 +0000 UTC m=+0.371332890 container start 3db686fdc65a4dd2459563d4e4b48ede1502406f34dfe56edb8cf5b756696e94 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T13:02:05.651 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 podman[117653]: 2024-03-21 13:02:05.489734062 +0000 UTC m=+0.468201944 container attach 3db686fdc65a4dd2459563d4e4b48ede1502406f34dfe56edb8cf5b756696e94 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T13:02:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:06 smithi172 ceph-mon[84635]: pgmap v168: 161 pgs: 19 active+undersized+degraded, 13 peering, 2 stale+active+clean, 27 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-03-21T13:02:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:06 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 62/627 objects degraded (9.888%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:06 smithi172 ceph-mon[90110]: pgmap v168: 161 pgs: 19 active+undersized+degraded, 13 peering, 2 stale+active+clean, 27 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-03-21T13:02:06.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:06 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 62/627 objects degraded (9.888%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T13:02:06.147 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 bash[117653]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate[117668]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-21T13:02:06.148 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:05 smithi172 podman[117653]: 2024-03-21 13:02:05.893230495 +0000 UTC m=+0.871698375 container died 3db686fdc65a4dd2459563d4e4b48ede1502406f34dfe56edb8cf5b756696e94 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:02:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:06 smithi177 ceph-mon[67966]: pgmap v168: 161 pgs: 19 active+undersized+degraded, 13 peering, 2 stale+active+clean, 27 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 62/627 objects degraded (9.888%) 2024-03-21T13:02:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:06 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 62/627 objects degraded (9.888%), 19 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:06.856 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:06 smithi172 podman[117792]: 2024-03-21 13:02:06.502044228 +0000 UTC m=+0.599219049 container remove 3db686fdc65a4dd2459563d4e4b48ede1502406f34dfe56edb8cf5b756696e94 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T13:02:06.856 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:06 smithi172 podman[117854]: 2024-03-21 13:02:06.617057766 +0000 UTC m=+0.018251615 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:02:06.856 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:06 smithi172 podman[117854]: 2024-03-21 13:02:06.744962921 +0000 UTC m=+0.146156766 container create 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T13:02:07.146 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:06 smithi172 podman[117854]: 2024-03-21 13:02:06.955648314 +0000 UTC m=+0.356842158 container init 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-21T13:02:07.146 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:06 smithi172 podman[117854]: 2024-03-21 13:02:06.958614172 +0000 UTC m=+0.359808019 container start 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:02:07.146 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:06 smithi172 ceph-osd[117872]: -- 172.21.15.172:0/3273295701 <== mon.1 v2:172.21.15.172:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558145f81680 con 0x558145f4b000 2024-03-21T13:02:07.146 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:07 smithi172 bash[117854]: 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 2024-03-21T13:02:07.146 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:07 smithi172 systemd[1]: Started Ceph osd.3 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:02:07.893 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:07 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:02:07.506+0000 7f19781757c0 -1 Falling back to public interface 2024-03-21T13:02:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[90110]: pgmap v169: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:08.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:08.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:08.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:08.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:08 smithi172 ceph-mon[84635]: pgmap v169: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:08 smithi177 ceph-mon[67966]: pgmap v169: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:09 smithi172 ceph-mon[84635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-21T13:02:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:09 smithi172 ceph-mon[90110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-21T13:02:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:09 smithi177 ceph-mon[67966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-03-21T13:02:10.341 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:10 smithi172 ceph-mon[84635]: pgmap v170: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:10.342 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:10 smithi172 ceph-mon[90110]: pgmap v170: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:10.342 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:10 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:02:10.209+0000 7f19781757c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-21T13:02:10.342 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:10 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:02:10.239+0000 7f19781757c0 -1 osd.3 121 log_to_monitors true 2024-03-21T13:02:10.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:10 smithi177 ceph-mon[67966]: pgmap v170: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:11 smithi177 ceph-mon[67966]: from='osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:02:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:11 smithi177 ceph-mon[67966]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:02:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-mon[84635]: from='osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:02:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-mon[84635]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:02:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:11.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-mon[90110]: from='osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:02:11.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-mon[90110]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T13:02:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:11.397 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:02:11 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:02:11.102+0000 7f19695cd700 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: osdmap e124: 8 total, 7 up, 8 in 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: from='osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: pgmap v172: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:12 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: osdmap e124: 8 total, 7 up, 8 in 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: from='osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: pgmap v172: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:12.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T13:02:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: osdmap e124: 8 total, 7 up, 8 in 2024-03-21T13:02:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: from='osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:02:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-21T13:02:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: pgmap v172: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 80/627 objects degraded (12.759%) 2024-03-21T13:02:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:12 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356] boot 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:13 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:02:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356] boot 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: osd.3 [v2:172.21.15.172:6826/987074356,v1:172.21.15.172:6827/987074356] boot 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:13 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 80/627 objects degraded (12.759%), 25 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:14.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:14 smithi172 ceph-mon[84635]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T13:02:14.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:14 smithi172 ceph-mon[84635]: pgmap v175: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-03-21T13:02:14.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:14 smithi172 ceph-mon[90110]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T13:02:14.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:14 smithi172 ceph-mon[90110]: pgmap v175: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-03-21T13:02:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:14 smithi177 ceph-mon[67966]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T13:02:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:14 smithi177 ceph-mon[67966]: pgmap v175: 161 pgs: 25 active+undersized+degraded, 41 active+undersized, 95 active+clean; 585 KiB data, 172 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 80/627 objects degraded (12.759%) 2024-03-21T13:02:15.630 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:02:15] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-21T13:02:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[90110]: pgmap v176: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 46/627 objects degraded (7.337%) 2024-03-21T13:02:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:02:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[84635]: pgmap v176: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 46/627 objects degraded (7.337%) 2024-03-21T13:02:16.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:16.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:02:16.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:16 smithi177 ceph-mon[67966]: pgmap v176: 161 pgs: 15 active+undersized+degraded, 25 active+undersized, 121 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 46/627 objects degraded (7.337%) 2024-03-21T13:02:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:02:16.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[90110]: pgmap v177: 161 pgs: 4 active+undersized+degraded, 9 active+undersized, 148 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 10/627 objects degraded (1.595%) 2024-03-21T13:02:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[90110]: pgmap v178: 161 pgs: 4 active+undersized+degraded, 9 active+undersized, 148 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 10/627 objects degraded (1.595%) 2024-03-21T13:02:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[84635]: pgmap v177: 161 pgs: 4 active+undersized+degraded, 9 active+undersized, 148 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 10/627 objects degraded (1.595%) 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[84635]: pgmap v178: 161 pgs: 4 active+undersized+degraded, 9 active+undersized, 148 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 10/627 objects degraded (1.595%) 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:02:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:17 smithi177 ceph-mon[67966]: pgmap v177: 161 pgs: 4 active+undersized+degraded, 9 active+undersized, 148 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 10/627 objects degraded (1.595%) 2024-03-21T13:02:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:17 smithi177 ceph-mon[67966]: pgmap v178: 161 pgs: 4 active+undersized+degraded, 9 active+undersized, 148 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 10/627 objects degraded (1.595%) 2024-03-21T13:02:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:17 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:02:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:17 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:02:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:17 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:17.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:17 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:02:18.146 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:02:17.929+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.147 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:18.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:02:18.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:17 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:02:17.941+0000 7f0d06c0d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.608 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.609 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:18.610 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.611 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.612 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:18.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:18.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T13:02:18.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 10/627 objects degraded (1.595%), 4 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:18.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:18.613 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.646 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.647 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:02:18.648 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.649 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.650 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.651 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:18.652 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:18.653 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:18.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 10/627 objects degraded (1.595%), 4 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:18.655 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:18.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: Upgrade: unsafe to stop osd(s) at this time (5 PGs are or would become offline) 2024-03-21T13:02:18.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 10/627 objects degraded (1.595%), 4 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:18.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:18.656 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:19.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:19 smithi177 ceph-mon[67966]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:19.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:19 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/627 objects degraded (1.595%), 4 pgs degraded) 2024-03-21T13:02:19.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:19 smithi172 ceph-mon[84635]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:19.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:19 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/627 objects degraded (1.595%), 4 pgs degraded) 2024-03-21T13:02:19.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:19 smithi172 ceph-mon[90110]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:19.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:19 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 10/627 objects degraded (1.595%), 4 pgs degraded) 2024-03-21T13:02:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:21 smithi177 ceph-mon[67966]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:21.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:21 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:21.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:21 smithi172 ceph-mon[84635]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:21.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:21 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:21 smithi172 ceph-mon[90110]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:21.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:21 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:23.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:23 smithi177 ceph-mon[67966]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s 2024-03-21T13:02:23.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:23 smithi172 ceph-mon[84635]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s 2024-03-21T13:02:23.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:23 smithi172 ceph-mon[90110]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s 2024-03-21T13:02:25.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:25 smithi177 ceph-mon[67966]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s 2024-03-21T13:02:25.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:25 smithi172 ceph-mon[84635]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s 2024-03-21T13:02:25.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:25 smithi172 ceph-mon[90110]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s 2024-03-21T13:02:25.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:02:25] "GET /metrics HTTP/1.1" 200 38197 "" "Prometheus/2.43.0" 2024-03-21T13:02:26.601 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:02:27.254 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:02:27.254 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (7m) 15s ago 29m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:02:27.254 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (6m) 4m ago 28m 79.7M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:02:27.254 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 15s ago 28m 68.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:02:27.254 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (8m) 4m ago 33m 448M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (18m) 15s ago 34m 509M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (4m) 15s ago 35m 53.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (5m) 4m ago 33m 35.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (4m) 15s ago 33m 41.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (7m) 15s ago 29m 15.6M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (7m) 4m ago 29m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (2m) 15s ago 33m 52.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (2m) 15s ago 32m 51.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (75s) 15s ago 32m 41.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (20s) 15s ago 31m 14.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (31m) 4m ago 31m 55.1M 3248M 17.2.0 e1d6a67b021e 49a2b745d53e 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (31m) 4m ago 31m 51.8M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (30m) 4m ago 30m 53.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (30m) 4m ago 30m 55.9M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:02:27.255 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (7m) 4m ago 29m 62.1M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:02:27.256 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (28m) 15s ago 28m 78.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:02:27.256 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (28m) 4m ago 28m 79.2M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:02:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:27 smithi177 ceph-mon[67966]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:02:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:27 smithi177 ceph-mon[67966]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:27.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:27 smithi177 ceph-mon[67966]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:27 smithi172 ceph-mon[90110]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:02:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:27 smithi172 ceph-mon[90110]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:27.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:27 smithi172 ceph-mon[90110]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:27.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:27 smithi172 ceph-mon[84635]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:02:27.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:27 smithi172 ceph-mon[84635]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:27.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:27 smithi172 ceph-mon[84635]: from='client.64414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:27.649 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:02:27.649 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:02:27.649 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:02:27.649 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 4 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 9 2024-03-21T13:02:27.650 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:02:27.651 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "4/8 daemons upgraded", 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:02:27.979 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:02:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:28 smithi177 ceph-mon[67966]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:28 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/712404476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:28 smithi177 ceph-mon[67966]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:28.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[84635]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:28.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/712404476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:28.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[84635]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:28.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[90110]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/712404476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[90110]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:28.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:29.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:29 smithi177 ceph-mon[67966]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:29.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:29 smithi172 ceph-mon[84635]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:29 smithi172 ceph-mon[90110]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:31.402 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:31 smithi172 ceph-mon[84635]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:31.402 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:31.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:31 smithi172 ceph-mon[90110]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:31.402 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:31.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:31 smithi177 ceph-mon[67966]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:31.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:33 smithi177 ceph-mon[67966]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:33 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:33 smithi177 ceph-mon[67966]: Upgrade: osd.4 is safe to restart 2024-03-21T13:02:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:33.538 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[90110]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:33.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[90110]: Upgrade: osd.4 is safe to restart 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[84635]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[84635]: Upgrade: osd.4 is safe to restart 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:33.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:35.629 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:02:35] "GET /metrics HTTP/1.1" 200 38206 "" "Prometheus/2.43.0" 2024-03-21T13:02:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:35 smithi177 ceph-mon[67966]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:35 smithi177 ceph-mon[67966]: Upgrade: Updating osd.4 2024-03-21T13:02:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T13:02:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:35 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:35.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:35 smithi177 ceph-mon[67966]: Deploying daemon osd.4 on smithi177 2024-03-21T13:02:35.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[84635]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:35.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[84635]: Upgrade: Updating osd.4 2024-03-21T13:02:35.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:35.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T13:02:35.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:35.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[84635]: Deploying daemon osd.4 on smithi177 2024-03-21T13:02:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[90110]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[90110]: Upgrade: Updating osd.4 2024-03-21T13:02:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T13:02:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:35.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:35 smithi172 ceph-mon[90110]: Deploying daemon osd.4 on smithi177 2024-03-21T13:02:37.812 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:37 smithi177 ceph-mon[67966]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:37.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:37 smithi172 ceph-mon[84635]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:37.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:37 smithi172 ceph-mon[90110]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:38.100 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:37 smithi177 systemd[1]: Stopping Ceph osd.4 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:02:38.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[29147]: 2024-03-21T13:02:38.099+0000 7f3b8c027700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:02:38.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[29147]: 2024-03-21T13:02:38.099+0000 7f3b8c027700 -1 osd.4 126 *** Got signal Terminated *** 2024-03-21T13:02:38.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[29147]: 2024-03-21T13:02:38.099+0000 7f3b8c027700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:02:38.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:38 smithi172 ceph-mon[84635]: osd.4 marked itself down and dead 2024-03-21T13:02:38.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:38 smithi172 ceph-mon[90110]: osd.4 marked itself down and dead 2024-03-21T13:02:38.980 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:38 smithi177 ceph-mon[67966]: osd.4 marked itself down and dead 2024-03-21T13:02:39.275 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:39 smithi177 podman[77820]: 2024-03-21 13:02:39.025464498 +0000 UTC m=+1.047837212 container died 49a2b745d53e30810280158cb5324b464a465399a34e4a8c299544b5209ff973 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, vcs-type=git, io.openshift.expose-services=, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_BRANCH=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.buildah.version=1.19.8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vendor=Red Hat, Inc.) 2024-03-21T13:02:39.567 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:39 smithi177 podman[77820]: 2024-03-21 13:02:39.366020314 +0000 UTC m=+1.388393073 container cleanup 49a2b745d53e30810280158cb5324b464a465399a34e4a8c299544b5209ff973 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, version=8, io.openshift.expose-services=, vendor=Red Hat, Inc., ceph=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_BRANCH=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, distribution-scope=public) 2024-03-21T13:02:39.567 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:39 smithi177 bash[77820]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4 2024-03-21T13:02:39.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:39 smithi177 podman[77850]: 2024-03-21 13:02:39.567189332 +0000 UTC m=+0.542177262 container remove 49a2b745d53e30810280158cb5324b464a465399a34e4a8c299544b5209ff973 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0) 2024-03-21T13:02:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:39 smithi177 ceph-mon[67966]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:39 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:02:39.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:39 smithi177 ceph-mon[67966]: osdmap e127: 8 total, 7 up, 8 in 2024-03-21T13:02:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:39 smithi172 ceph-mon[84635]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:39 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:02:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:39 smithi172 ceph-mon[84635]: osdmap e127: 8 total, 7 up, 8 in 2024-03-21T13:02:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:39 smithi172 ceph-mon[90110]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:39 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:02:40.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:39 smithi172 ceph-mon[90110]: osdmap e127: 8 total, 7 up, 8 in 2024-03-21T13:02:40.287 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:39 smithi177 podman[77941]: 2024-03-21 13:02:39.890228307 +0000 UTC m=+0.018106960 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:02:40.287 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:39 smithi177 podman[77941]: 2024-03-21 13:02:39.993455917 +0000 UTC m=+0.121334570 container create e0dcc45e81a6919cf41f2e1cef2680cce284e67561a9e1f660bd959581d6430a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-21T13:02:40.287 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:40 smithi177 podman[77941]: 2024-03-21 13:02:40.204409216 +0000 UTC m=+0.332287869 container init e0dcc45e81a6919cf41f2e1cef2680cce284e67561a9e1f660bd959581d6430a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T13:02:40.287 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:40 smithi177 podman[77941]: 2024-03-21 13:02:40.208031942 +0000 UTC m=+0.335910595 container start e0dcc45e81a6919cf41f2e1cef2680cce284e67561a9e1f660bd959581d6430a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:02:40.287 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:40 smithi177 podman[77941]: 2024-03-21 13:02:40.28745168 +0000 UTC m=+0.415330348 container attach e0dcc45e81a6919cf41f2e1cef2680cce284e67561a9e1f660bd959581d6430a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T13:02:40.608 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:40 smithi177 podman[77941]: 2024-03-21 13:02:40.503324601 +0000 UTC m=+0.631203264 container died e0dcc45e81a6919cf41f2e1cef2680cce284e67561a9e1f660bd959581d6430a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:02:41.099 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:40 smithi177 ceph-mon[67966]: osdmap e128: 8 total, 7 up, 8 in 2024-03-21T13:02:41.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:40 smithi172 ceph-mon[84635]: osdmap e128: 8 total, 7 up, 8 in 2024-03-21T13:02:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:40 smithi172 ceph-mon[90110]: osdmap e128: 8 total, 7 up, 8 in 2024-03-21T13:02:41.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 podman[77967]: 2024-03-21 13:02:41.099490778 +0000 UTC m=+0.590847589 container remove e0dcc45e81a6919cf41f2e1cef2680cce284e67561a9e1f660bd959581d6430a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:02:41.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-21T13:02:41.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 systemd[1]: Stopped Ceph osd.4 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:02:41.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.4.service: Consumed 16.518s CPU time. 2024-03-21T13:02:41.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 systemd[1]: Starting Ceph osd.4 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:02:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:41 smithi177 ceph-mon[67966]: pgmap v192: 161 pgs: 32 peering, 129 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:41 smithi177 ceph-mon[67966]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-21T13:02:41.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:41 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:41.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 podman[78069]: 2024-03-21 13:02:41.438457547 +0000 UTC m=+0.018013124 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:02:41.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 podman[78069]: 2024-03-21 13:02:41.550905443 +0000 UTC m=+0.130461055 container create 2b325db51d1b17a9d29a47774d0ad6b8a9f6b4b91045f0d27044a67db1869b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:02:41.859 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 podman[78069]: 2024-03-21 13:02:41.778360384 +0000 UTC m=+0.357915968 container init 2b325db51d1b17a9d29a47774d0ad6b8a9f6b4b91045f0d27044a67db1869b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-21T13:02:41.859 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 podman[78069]: 2024-03-21 13:02:41.781557986 +0000 UTC m=+0.361113557 container start 2b325db51d1b17a9d29a47774d0ad6b8a9f6b4b91045f0d27044a67db1869b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-21T13:02:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:41 smithi172 ceph-mon[84635]: pgmap v192: 161 pgs: 32 peering, 129 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:42.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:41 smithi172 ceph-mon[84635]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-21T13:02:42.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:41 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:41 smithi172 ceph-mon[90110]: pgmap v192: 161 pgs: 32 peering, 129 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:02:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:41 smithi172 ceph-mon[90110]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-21T13:02:42.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:41 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:42.253 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:41 smithi177 podman[78069]: 2024-03-21 13:02:41.896648181 +0000 UTC m=+0.476203769 container attach 2b325db51d1b17a9d29a47774d0ad6b8a9f6b4b91045f0d27044a67db1869b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T13:02:42.590 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 bash[78069]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate[78084]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-21T13:02:42.591 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 podman[78069]: 2024-03-21 13:02:42.278539614 +0000 UTC m=+0.858095190 container died 2b325db51d1b17a9d29a47774d0ad6b8a9f6b4b91045f0d27044a67db1869b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T13:02:42.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:42 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 24/627 objects degraded (3.828%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:43.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:42 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 24/627 objects degraded (3.828%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:43.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:42 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 24/627 objects degraded (3.828%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:43.263 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:42 smithi177 podman[78212]: 2024-03-21 13:02:42.915616526 +0000 UTC m=+0.631979725 container remove 2b325db51d1b17a9d29a47774d0ad6b8a9f6b4b91045f0d27044a67db1869b10 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:02:43.263 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 podman[78275]: 2024-03-21 13:02:43.029576685 +0000 UTC m=+0.017990430 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:02:43.264 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 podman[78275]: 2024-03-21 13:02:43.141629943 +0000 UTC m=+0.130043676 container create 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-21T13:02:43.710 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 podman[78275]: 2024-03-21 13:02:43.386782306 +0000 UTC m=+0.375196043 container init 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:02:43.710 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 podman[78275]: 2024-03-21 13:02:43.389794901 +0000 UTC m=+0.378208637 container start 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-21T13:02:43.710 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 ceph-osd[78297]: -- 172.21.15.177:0/295404644 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605cfec1680 con 0x5605cfe8ac00 2024-03-21T13:02:43.711 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 bash[78275]: 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac 2024-03-21T13:02:43.711 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 systemd[1]: Started Ceph osd.4 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:02:43.711 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:43 smithi177 ceph-mon[67966]: pgmap v193: 161 pgs: 8 active+undersized+degraded, 32 peering, 19 active+undersized, 102 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-03-21T13:02:43.711 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:43 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:43.711 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:43 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:43.711 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:43 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:44.108 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:02:43.938+0000 7fb1d73747c0 -1 Falling back to public interface 2024-03-21T13:02:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[84635]: pgmap v193: 161 pgs: 8 active+undersized+degraded, 32 peering, 19 active+undersized, 102 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-03-21T13:02:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[90110]: pgmap v193: 161 pgs: 8 active+undersized+degraded, 32 peering, 19 active+undersized, 102 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 24/627 objects degraded (3.828%) 2024-03-21T13:02:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:43 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:02:45] "GET /metrics HTTP/1.1" 200 38206 "" "Prometheus/2.43.0" 2024-03-21T13:02:45.969 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:45 smithi177 ceph-mon[67966]: pgmap v194: 161 pgs: 22 active+undersized+degraded, 9 peering, 37 active+undersized, 93 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-21T13:02:46.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:45 smithi172 ceph-mon[84635]: pgmap v194: 161 pgs: 22 active+undersized+degraded, 9 peering, 37 active+undersized, 93 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-21T13:02:46.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:45 smithi172 ceph-mon[90110]: pgmap v194: 161 pgs: 22 active+undersized+degraded, 9 peering, 37 active+undersized, 93 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 82/627 objects degraded (13.078%) 2024-03-21T13:02:46.933 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:02:46.638+0000 7fb1d73747c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-21T13:02:46.933 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:02:46.671+0000 7fb1d73747c0 -1 osd.4 126 log_to_monitors true 2024-03-21T13:02:46.933 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:46 smithi177 ceph-mon[67966]: from='osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:02:46.933 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:46 smithi177 ceph-mon[67966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:02:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:46 smithi172 ceph-mon[84635]: from='osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:02:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:46 smithi172 ceph-mon[84635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:02:47.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:46 smithi172 ceph-mon[90110]: from='osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:02:47.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:46 smithi172 ceph-mon[90110]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T13:02:47.543 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:02:47.194+0000 7fb1c47c4700 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: pgmap v195: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: osdmap e129: 8 total, 7 up, 8 in 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: from='osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:02:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:47 smithi177 ceph-mon[67966]: from='osd.4 ' entity='osd.4' 2024-03-21T13:02:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: pgmap v195: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-03-21T13:02:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-21T13:02:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T13:02:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: osdmap e129: 8 total, 7 up, 8 in 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: from='osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[84635]: from='osd.4 ' entity='osd.4' 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: pgmap v195: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 177 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: osdmap e129: 8 total, 7 up, 8 in 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: from='osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:02:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:47 smithi172 ceph-mon[90110]: from='osd.4 ' entity='osd.4' 2024-03-21T13:02:49.052 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:49.052 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.052 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:49.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:02:49.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073] boot 2024-03-21T13:02:49.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T13:02:49.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:02:49.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:49.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073] boot 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 106/627 objects degraded (16.906%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:02:49.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: osd.4 [v2:172.21.15.177:6800/3498368073,v1:172.21.15.177:6801/3498368073] boot 2024-03-21T13:02:49.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T13:02:49.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T13:02:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:50 smithi177 ceph-mon[67966]: pgmap v197: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-03-21T13:02:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:50 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:50 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:50 smithi177 ceph-mon[67966]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T13:02:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[84635]: pgmap v197: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-03-21T13:02:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[84635]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T13:02:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[90110]: pgmap v197: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 106/627 objects degraded (16.906%) 2024-03-21T13:02:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:50 smithi172 ceph-mon[90110]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T13:02:52.261 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:52 smithi177 ceph-mon[67966]: pgmap v200: 161 pgs: 30 peering, 13 active+undersized+degraded, 25 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-03-21T13:02:52.262 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:52 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:52 smithi172 ceph-mon[84635]: pgmap v200: 161 pgs: 30 peering, 13 active+undersized+degraded, 25 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-03-21T13:02:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:52 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:52 smithi172 ceph-mon[90110]: pgmap v200: 161 pgs: 30 peering, 13 active+undersized+degraded, 25 active+undersized, 93 active+clean; 585 KiB data, 188 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 40/627 objects degraded (6.380%) 2024-03-21T13:02:52.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:52 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: pgmap v201: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: pgmap v202: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: pgmap v203: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:02:53.638 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:53.639 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:02:53.639 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:02:53.639 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:53.639 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:02:53.639 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 40/627 objects degraded (6.380%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:53.639 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 13 pgs degraded) 2024-03-21T13:02:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: pgmap v201: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:53.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: pgmap v202: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: pgmap v203: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 40/627 objects degraded (6.380%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:53.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:53 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 13 pgs degraded) 2024-03-21T13:02:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: pgmap v201: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:02:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:02:54.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: pgmap v202: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: pgmap v203: 161 pgs: 30 peering, 131 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 40/627 objects degraded (6.380%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T13:02:54.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:53 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 40/627 objects degraded (6.380%), 13 pgs degraded) 2024-03-21T13:02:54.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:02:54.702+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:54.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:54.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:02:54.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:02:54.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:54.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:54 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:02:55] "GET /metrics HTTP/1.1" 200 38054 "" "Prometheus/2.43.0" 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:55.858 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:55.859 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:55.860 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:55.861 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T13:02:55.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:55 smithi177 ceph-mon[67966]: Upgrade: osd.5 is safe to restart 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:56.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:56.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:56.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:56.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 792 B/s rd, 0 op/s 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:56.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:02:56.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:02:56.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:02:56.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T13:02:56.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[84635]: Upgrade: osd.5 is safe to restart 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-21T13:02:56.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:55 smithi172 ceph-mon[90110]: Upgrade: osd.5 is safe to restart 2024-03-21T13:02:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:57 smithi177 ceph-mon[67966]: Upgrade: Updating osd.5 2024-03-21T13:02:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T13:02:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:57 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:57 smithi177 ceph-mon[67966]: Deploying daemon osd.5 on smithi177 2024-03-21T13:02:57.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:57 smithi177 ceph-mon[67966]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:02:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[84635]: Upgrade: Updating osd.5 2024-03-21T13:02:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T13:02:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[84635]: Deploying daemon osd.5 on smithi177 2024-03-21T13:02:57.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[84635]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:02:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[90110]: Upgrade: Updating osd.5 2024-03-21T13:02:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:02:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T13:02:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:02:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[90110]: Deploying daemon osd.5 on smithi177 2024-03-21T13:02:57.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:57 smithi172 ceph-mon[90110]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:02:58.346 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:02:58.996 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (7m) 47s ago 29m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (7m) 11s ago 29m 81.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 47s ago 28m 68.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (8m) 11s ago 33m 453M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (18m) 47s ago 35m 509M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (5m) 47s ago 35m 53.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (6m) 11s ago 34m 45.3M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (4m) 47s ago 34m 41.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (8m) 47s ago 30m 15.6M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (8m) 11s ago 30m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (3m) 47s ago 33m 52.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (2m) 47s ago 33m 51.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:02:58.997 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (106s) 47s ago 32m 41.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (52s) 47s ago 32m 14.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (15s) 11s ago 32m 14.1M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (31m) 11s ago 31m 55.3M 3248M 17.2.0 e1d6a67b021e e6b3d6ec3e28 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (31m) 11s ago 31m 56.4M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (30m) 11s ago 30m 60.3M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (7m) 11s ago 30m 75.4M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (28m) 47s ago 28m 78.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:02:58.998 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (28m) 11s ago 28m 80.5M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:02:59.393 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:02:59.393 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:02:59.393 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:02:59.393 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:59.393 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:02:59.393 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 5 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 10 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:02:59.394 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:02:59.724 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "5/8 daemons upgraded", 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:02:59.725 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:02:59.858 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:02:59 smithi177 systemd[1]: Stopping Ceph osd.5 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:02:59.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:59 smithi177 ceph-mon[67966]: from='client.64444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:59.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:59 smithi177 ceph-mon[67966]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:02:59.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:59 smithi177 ceph-mon[67966]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:59.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:59 smithi177 ceph-mon[67966]: from='client.64456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:02:59.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:02:59 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/222350170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[84635]: from='client.64444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[84635]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[84635]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[84635]: from='client.64456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:00.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/222350170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[90110]: from='client.64444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[90110]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[90110]: from='client.54516 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[90110]: from='client.64456 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:00.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:02:59 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/222350170' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:00.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[32089]: 2024-03-21T13:03:00.030+0000 7f68c76af700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:03:00.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[32089]: 2024-03-21T13:03:00.030+0000 7f68c76af700 -1 osd.5 131 *** Got signal Terminated *** 2024-03-21T13:03:00.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[32089]: 2024-03-21T13:03:00.030+0000 7f68c76af700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:03:01.048 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:00 smithi172 ceph-mon[84635]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:01.049 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:00 smithi172 ceph-mon[84635]: osd.5 marked itself down and dead 2024-03-21T13:03:01.049 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:00 smithi172 ceph-mon[90110]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:01.049 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:00 smithi172 ceph-mon[90110]: osd.5 marked itself down and dead 2024-03-21T13:03:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:00 smithi177 ceph-mon[67966]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:01.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:00 smithi177 ceph-mon[67966]: osd.5 marked itself down and dead 2024-03-21T13:03:01.362 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:01 smithi177 podman[82645]: 2024-03-21 13:03:01.174740932 +0000 UTC m=+1.267141237 container died e6b3d6ec3e28af1719727d58d8f7c1a2f794dca06421dfadf6cc359c83ec273a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., version=8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_CLEAN=True, release=754, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0) 2024-03-21T13:03:01.710 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:01 smithi177 podman[82645]: 2024-03-21 13:03:01.513849862 +0000 UTC m=+1.606250181 container cleanup e6b3d6ec3e28af1719727d58d8f7c1a2f794dca06421dfadf6cc359c83ec273a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, distribution-scope=public, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD) 2024-03-21T13:03:01.710 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:01 smithi177 bash[82645]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[84635]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 601 B/s rd, 0 op/s 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[84635]: osdmap e132: 8 total, 7 up, 8 in 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[90110]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 601 B/s rd, 0 op/s 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[90110]: osdmap e132: 8 total, 7 up, 8 in 2024-03-21T13:03:02.065 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:01 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:01 smithi177 ceph-mon[67966]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 601 B/s rd, 0 op/s 2024-03-21T13:03:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:01 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:03:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:01 smithi177 ceph-mon[67966]: osdmap e132: 8 total, 7 up, 8 in 2024-03-21T13:03:02.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:02.108 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:01 smithi177 podman[82663]: 2024-03-21 13:03:01.723439282 +0000 UTC m=+0.549440951 container remove e6b3d6ec3e28af1719727d58d8f7c1a2f794dca06421dfadf6cc359c83ec273a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vendor=Red Hat, Inc., ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, version=8, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD) 2024-03-21T13:03:02.385 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:02 smithi177 podman[82765]: 2024-03-21 13:03:02.13304758 +0000 UTC m=+0.109046884 container create 80a6db4e25a7d896ede3e999044a96c3c08db995f3c55b4279104315c7f0be57 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:03:02.385 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:02 smithi177 podman[82765]: 2024-03-21 13:03:02.044797105 +0000 UTC m=+0.020796415 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:03:02.386 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:02 smithi177 podman[82765]: 2024-03-21 13:03:02.386006601 +0000 UTC m=+0.362005906 container init 80a6db4e25a7d896ede3e999044a96c3c08db995f3c55b4279104315c7f0be57 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-21T13:03:02.679 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:02 smithi177 podman[82765]: 2024-03-21 13:03:02.389270921 +0000 UTC m=+0.365270232 container start 80a6db4e25a7d896ede3e999044a96c3c08db995f3c55b4279104315c7f0be57 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:03:02.679 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:02 smithi177 podman[82765]: 2024-03-21 13:03:02.494225774 +0000 UTC m=+0.470225081 container attach 80a6db4e25a7d896ede3e999044a96c3c08db995f3c55b4279104315c7f0be57 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T13:03:03.039 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:02 smithi177 ceph-mon[67966]: osdmap e133: 8 total, 7 up, 8 in 2024-03-21T13:03:03.039 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:02 smithi177 podman[82765]: 2024-03-21 13:03:02.679695987 +0000 UTC m=+0.655695291 container died 80a6db4e25a7d896ede3e999044a96c3c08db995f3c55b4279104315c7f0be57 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:03:03.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:02 smithi172 ceph-mon[84635]: osdmap e133: 8 total, 7 up, 8 in 2024-03-21T13:03:03.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:02 smithi172 ceph-mon[90110]: osdmap e133: 8 total, 7 up, 8 in 2024-03-21T13:03:03.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 podman[82792]: 2024-03-21 13:03:03.315075669 +0000 UTC m=+0.627225023 container remove 80a6db4e25a7d896ede3e999044a96c3c08db995f3c55b4279104315c7f0be57 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-21T13:03:03.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-21T13:03:03.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 systemd[1]: Stopped Ceph osd.5 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:03:03.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.5.service: Consumed 16.454s CPU time. 2024-03-21T13:03:03.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 systemd[1]: Starting Ceph osd.5 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:03:04.052 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:03 smithi177 ceph-mon[67966]: pgmap v210: 161 pgs: 7 active+undersized+degraded, 10 peering, 7 stale+active+clean, 20 active+undersized, 117 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 27/627 objects degraded (4.306%) 2024-03-21T13:03:04.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:03 smithi177 ceph-mon[67966]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-21T13:03:04.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:03 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 27/627 objects degraded (4.306%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:04.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:04.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:04.053 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:04.053 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 podman[82897]: 2024-03-21 13:03:03.666561932 +0000 UTC m=+0.017591946 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:03:04.053 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:03 smithi177 podman[82897]: 2024-03-21 13:03:03.783114504 +0000 UTC m=+0.134144555 container create eae0d86b25c37342fd3511c06256bedfc35eb12410e637beebdb2973d7db8157 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T13:03:04.053 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 podman[82897]: 2024-03-21 13:03:04.052978821 +0000 UTC m=+0.404008843 container init eae0d86b25c37342fd3511c06256bedfc35eb12410e637beebdb2973d7db8157 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:03:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[84635]: pgmap v210: 161 pgs: 7 active+undersized+degraded, 10 peering, 7 stale+active+clean, 20 active+undersized, 117 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 27/627 objects degraded (4.306%) 2024-03-21T13:03:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[84635]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-21T13:03:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 27/627 objects degraded (4.306%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:04.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[90110]: pgmap v210: 161 pgs: 7 active+undersized+degraded, 10 peering, 7 stale+active+clean, 20 active+undersized, 117 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 27/627 objects degraded (4.306%) 2024-03-21T13:03:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[90110]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-21T13:03:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 27/627 objects degraded (4.306%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:04.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:04.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 podman[82897]: 2024-03-21 13:03:04.056604482 +0000 UTC m=+0.407634497 container start eae0d86b25c37342fd3511c06256bedfc35eb12410e637beebdb2973d7db8157 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-21T13:03:04.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 podman[82897]: 2024-03-21 13:03:04.144115735 +0000 UTC m=+0.495145757 container attach eae0d86b25c37342fd3511c06256bedfc35eb12410e637beebdb2973d7db8157 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T13:03:04.856 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:04.856 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:04.856 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:04.856 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:04.856 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T13:03:04.856 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 bash[82897]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate[82913]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-21T13:03:04.857 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:04 smithi177 podman[82897]: 2024-03-21 13:03:04.553858161 +0000 UTC m=+0.904888162 container died eae0d86b25c37342fd3511c06256bedfc35eb12410e637beebdb2973d7db8157 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:03:05.483 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 podman[83051]: 2024-03-21 13:03:05.148120419 +0000 UTC m=+0.589276953 container remove eae0d86b25c37342fd3511c06256bedfc35eb12410e637beebdb2973d7db8157 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:03:05.483 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 podman[83115]: 2024-03-21 13:03:05.267713532 +0000 UTC m=+0.017963528 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:03:05.483 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 podman[83115]: 2024-03-21 13:03:05.374355472 +0000 UTC m=+0.124605511 container create 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-21T13:03:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:03:05] "GET /metrics HTTP/1.1" 200 38281 "" "Prometheus/2.43.0" 2024-03-21T13:03:05.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:05 smithi177 ceph-mon[67966]: pgmap v211: 161 pgs: 18 active+undersized+degraded, 10 peering, 31 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-03-21T13:03:05.858 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 podman[83115]: 2024-03-21 13:03:05.577116194 +0000 UTC m=+0.327366204 container init 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:03:05.858 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 podman[83115]: 2024-03-21 13:03:05.580537953 +0000 UTC m=+0.330787950 container start 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:03:05.858 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 ceph-osd[83134]: -- 172.21.15.177:0/942386687 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5628be041680 con 0x5628be00b400 2024-03-21T13:03:05.858 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 bash[83115]: 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d 2024-03-21T13:03:05.859 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:05 smithi177 systemd[1]: Started Ceph osd.5 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:03:06.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:05 smithi172 ceph-mon[84635]: pgmap v211: 161 pgs: 18 active+undersized+degraded, 10 peering, 31 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-03-21T13:03:06.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:05 smithi172 ceph-mon[90110]: pgmap v211: 161 pgs: 18 active+undersized+degraded, 10 peering, 31 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 55/627 objects degraded (8.772%) 2024-03-21T13:03:06.547 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:03:06.124+0000 7f428061f7c0 -1 Falling back to public interface 2024-03-21T13:03:06.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:06.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:06.859 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:07.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:07.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:07.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:08.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:07 smithi177 ceph-mon[67966]: pgmap v212: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:08.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:07 smithi177 ceph-mon[67966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-21T13:03:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:07 smithi172 ceph-mon[84635]: pgmap v212: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:08.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:07 smithi172 ceph-mon[84635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-21T13:03:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:07 smithi172 ceph-mon[90110]: pgmap v212: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:08.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:07 smithi172 ceph-mon[90110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-21T13:03:09.048 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:08 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 23 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:09.048 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:03:08.843+0000 7f428061f7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-21T13:03:09.048 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:08 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:03:08.867+0000 7f428061f7c0 -1 osd.5 131 log_to_monitors true 2024-03-21T13:03:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:08 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 23 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:09.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:08 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 74/627 objects degraded (11.802%), 23 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:09.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:09 smithi177 ceph-mon[67966]: pgmap v213: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:09.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:09 smithi177 ceph-mon[67966]: from='osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:03:09.771 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:09 smithi177 ceph-mon[67966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:03:10.107 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:03:09 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:03:09.770+0000 7f4271a77700 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:03:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:09 smithi172 ceph-mon[84635]: pgmap v213: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:09 smithi172 ceph-mon[84635]: from='osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:03:10.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:09 smithi172 ceph-mon[84635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:03:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:09 smithi172 ceph-mon[90110]: pgmap v213: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:09 smithi172 ceph-mon[90110]: from='osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:03:10.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:09 smithi172 ceph-mon[90110]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T13:03:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:10 smithi177 ceph-mon[67966]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T13:03:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:10 smithi177 ceph-mon[67966]: from='osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:10 smithi177 ceph-mon[67966]: osdmap e134: 8 total, 7 up, 8 in 2024-03-21T13:03:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:10 smithi177 ceph-mon[67966]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:11.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[90110]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T13:03:11.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[90110]: from='osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[90110]: osdmap e134: 8 total, 7 up, 8 in 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[90110]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[84635]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[84635]: from='osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[84635]: osdmap e134: 8 total, 7 up, 8 in 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[84635]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:11.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.106 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: pgmap v215: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040] boot 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.107 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:11 smithi177 ceph-mon[67966]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T13:03:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: pgmap v215: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:03:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040] boot 2024-03-21T13:03:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T13:03:12.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[84635]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: pgmap v215: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 74/627 objects degraded (11.802%) 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: osd.5 [v2:172.21.15.177:6808/1469889040,v1:172.21.15.177:6809/1469889040] boot 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:12.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:11 smithi172 ceph-mon[90110]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T13:03:14.006 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:13 smithi177 ceph-mon[67966]: pgmap v218: 161 pgs: 4 activating, 5 peering, 18 active+undersized+degraded, 19 active+undersized, 115 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-03-21T13:03:14.007 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:13 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 18 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:13 smithi172 ceph-mon[84635]: pgmap v218: 161 pgs: 4 activating, 5 peering, 18 active+undersized+degraded, 19 active+undersized, 115 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-03-21T13:03:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:13 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 18 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:14.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:13 smithi172 ceph-mon[90110]: pgmap v218: 161 pgs: 4 activating, 5 peering, 18 active+undersized+degraded, 19 active+undersized, 115 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 56/627 objects degraded (8.931%) 2024-03-21T13:03:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:13 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 56/627 objects degraded (8.931%), 18 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:15.605 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:03:15] "GET /metrics HTTP/1.1" 200 38281 "" "Prometheus/2.43.0" 2024-03-21T13:03:15.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: pgmap v219: 161 pgs: 4 activating, 5 peering, 10 active+undersized+degraded, 12 active+undersized, 130 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 512 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: pgmap v220: 161 pgs: 4 activating, 5 peering, 10 active+undersized+degraded, 12 active+undersized, 130 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:15.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: pgmap v219: 161 pgs: 4 activating, 5 peering, 10 active+undersized+degraded, 12 active+undersized, 130 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 512 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: pgmap v220: 161 pgs: 4 activating, 5 peering, 10 active+undersized+degraded, 12 active+undersized, 130 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:15.898 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:15 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: pgmap v219: 161 pgs: 4 activating, 5 peering, 10 active+undersized+degraded, 12 active+undersized, 130 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 512 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: pgmap v220: 161 pgs: 4 activating, 5 peering, 10 active+undersized+degraded, 12 active+undersized, 130 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 626 B/s rd, 0 op/s; 34/627 objects degraded (5.423%) 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:03:16.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:16.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:15 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:03:16.507+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:16.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:16.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:03:16.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:16.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:16.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:16.652 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:03:16.518+0000 7f0d06c0d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-21T13:03:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:16 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:17.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:16 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:17 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:03:18.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:18.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:17 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:17 smithi177 ceph-mon[67966]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-21T13:03:18.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:17 smithi177 ceph-mon[67966]: pgmap v221: 161 pgs: 4 activating, 157 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:03:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:18.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:03:18.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:18.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:18.150 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[84635]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[84635]: pgmap v221: 161 pgs: 4 activating, 157 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:18.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:18.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[90110]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-21T13:03:18.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:17 smithi172 ceph-mon[90110]: pgmap v221: 161 pgs: 4 activating, 157 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:03:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:18 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-03-21T13:03:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:19.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:18 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-03-21T13:03:19.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:19.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:19.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:18 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/627 objects degraded (5.423%), 10 pgs degraded) 2024-03-21T13:03:19.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:19.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:20 smithi177 ceph-mon[67966]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:03:20.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:20 smithi172 ceph-mon[84635]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:03:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:20 smithi172 ceph-mon[90110]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:03:22.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:22 smithi177 ceph-mon[67966]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 575 B/s rd, 0 op/s 2024-03-21T13:03:22.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:22 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:22 smithi172 ceph-mon[84635]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 575 B/s rd, 0 op/s 2024-03-21T13:03:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:22 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:22 smithi172 ceph-mon[90110]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 575 B/s rd, 0 op/s 2024-03-21T13:03:22.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:22 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:24 smithi177 ceph-mon[67966]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:03:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:24 smithi172 ceph-mon[84635]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:03:24.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:24 smithi172 ceph-mon[90110]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:03:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:03:25] "GET /metrics HTTP/1.1" 200 38286 "" "Prometheus/2.43.0" 2024-03-21T13:03:26.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:26 smithi177 ceph-mon[67966]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:03:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:26 smithi172 ceph-mon[84635]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:03:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:26 smithi172 ceph-mon[90110]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:03:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:27 smithi177 ceph-mon[67966]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:27 smithi172 ceph-mon[84635]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:27 smithi172 ceph-mon[90110]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:29.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:29.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:29 smithi177 ceph-mon[67966]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:29.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:29.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:29 smithi172 ceph-mon[84635]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:29.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:29.650 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:29 smithi172 ceph-mon[90110]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:30.087 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (8m) 79s ago 30m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (7m) 20s ago 29m 79.1M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 79s ago 29m 68.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (9m) 20s ago 34m 453M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (19m) 79s ago 36m 509M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (5m) 79s ago 36m 53.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (6m) 20s ago 34m 46.5M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (5m) 79s ago 34m 41.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:03:30.739 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (8m) 79s ago 30m 15.6M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (8m) 20s ago 30m 15.2M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (3m) 79s ago 34m 52.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (3m) 79s ago 33m 51.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (2m) 79s ago 33m 41.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (84s) 79s ago 33m 14.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (47s) 20s ago 32m 52.1M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (25s) 20s ago 32m 12.5M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (31m) 20s ago 31m 57.1M 3248M 17.2.0 e1d6a67b021e 423e62a577d9 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (31m) 20s ago 31m 61.0M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (8m) 20s ago 30m 75.4M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (29m) 79s ago 29m 78.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:03:30.740 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (29m) 20s ago 29m 80.7M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:03:31.132 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 6 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 11 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:03:31.133 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:03:31.480 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:03:31.480 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:03:31.480 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:03:31.480 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T13:03:31.481 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:03:31.481 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "6/8 daemons upgraded", 2024-03-21T13:03:31.481 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T13:03:31.481 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:03:31.481 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:03:31.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: from='client.54549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/4286984806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: from='client.54549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:31.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:31.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:31.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/4286984806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:31.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: from='client.54549 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: from='client.64489 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/4286984806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:32 smithi177 ceph-mon[67966]: from='client.64501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:32 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:32 smithi177 ceph-mon[67966]: Upgrade: osd.6 is safe to restart 2024-03-21T13:03:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:32 smithi172 ceph-mon[84635]: from='client.64501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:32 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:32 smithi172 ceph-mon[84635]: Upgrade: osd.6 is safe to restart 2024-03-21T13:03:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:32 smithi172 ceph-mon[90110]: from='client.64501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:03:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:32 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-21T13:03:33.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:32 smithi172 ceph-mon[90110]: Upgrade: osd.6 is safe to restart 2024-03-21T13:03:34.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:33 smithi177 ceph-mon[67966]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:34.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:34.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:34.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T13:03:34.018 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:33 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[84635]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T13:03:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[90110]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T13:03:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:33 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:34.986 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:34 smithi177 ceph-mon[67966]: Upgrade: Updating osd.6 2024-03-21T13:03:34.987 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:34 smithi177 ceph-mon[67966]: Deploying daemon osd.6 on smithi177 2024-03-21T13:03:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:34 smithi172 ceph-mon[84635]: Upgrade: Updating osd.6 2024-03-21T13:03:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:34 smithi172 ceph-mon[84635]: Deploying daemon osd.6 on smithi177 2024-03-21T13:03:35.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:34 smithi172 ceph-mon[90110]: Upgrade: Updating osd.6 2024-03-21T13:03:35.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:34 smithi172 ceph-mon[90110]: Deploying daemon osd.6 on smithi177 2024-03-21T13:03:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:03:35] "GET /metrics HTTP/1.1" 200 38283 "" "Prometheus/2.43.0" 2024-03-21T13:03:36.097 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:35 smithi177 ceph-mon[67966]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:36.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:35 smithi172 ceph-mon[84635]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:36.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:35 smithi172 ceph-mon[90110]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:03:36.729 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:36 smithi177 systemd[1]: Stopping Ceph osd.6 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:03:36.729 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[34984]: 2024-03-21T13:03:36.709+0000 7fd69c487700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:03:36.729 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[34984]: 2024-03-21T13:03:36.709+0000 7fd69c487700 -1 osd.6 136 *** Got signal Terminated *** 2024-03-21T13:03:36.730 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[34984]: 2024-03-21T13:03:36.709+0000 7fd69c487700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:03:37.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:36 smithi177 ceph-mon[67966]: osd.6 marked itself down and dead 2024-03-21T13:03:37.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:36 smithi172 ceph-mon[90110]: osd.6 marked itself down and dead 2024-03-21T13:03:37.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:36 smithi172 ceph-mon[84635]: osd.6 marked itself down and dead 2024-03-21T13:03:37.535 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:37 smithi177 podman[87396]: 2024-03-21 13:03:37.240678004 +0000 UTC m=+0.638468914 container died 423e62a577d9a29ede10379ffbcd174a3e53b9afe167c0d09137083a830c9865 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=-17.2.0, release=754, maintainer=Guillaume Abrioux , architecture=x86_64, name=centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-03-21T13:03:37.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:37 smithi177 ceph-mon[67966]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:37.861 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:37 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:03:37.862 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:37 smithi177 ceph-mon[67966]: osdmap e137: 8 total, 7 up, 8 in 2024-03-21T13:03:37.862 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:37 smithi177 podman[87396]: 2024-03-21 13:03:37.760008295 +0000 UTC m=+1.157799208 container cleanup 423e62a577d9a29ede10379ffbcd174a3e53b9afe167c0d09137083a830c9865 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, release=754, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, name=centos-stream) 2024-03-21T13:03:37.862 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:37 smithi177 bash[87396]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6 2024-03-21T13:03:38.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:37 smithi172 ceph-mon[84635]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:38.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:37 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:03:38.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:37 smithi172 ceph-mon[84635]: osdmap e137: 8 total, 7 up, 8 in 2024-03-21T13:03:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:37 smithi172 ceph-mon[90110]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:03:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:37 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:03:38.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:37 smithi172 ceph-mon[90110]: osdmap e137: 8 total, 7 up, 8 in 2024-03-21T13:03:38.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:37 smithi177 podman[87413]: 2024-03-21 13:03:37.97001311 +0000 UTC m=+0.727151676 container remove 423e62a577d9a29ede10379ffbcd174a3e53b9afe167c0d09137083a830c9865 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vendor=Red Hat, Inc., RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public) 2024-03-21T13:03:38.693 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:38 smithi177 podman[87509]: 2024-03-21 13:03:38.36283809 +0000 UTC m=+0.101221938 container create 277fc9e56d205e71206198e6a89cd78ce03fb0e306300080e761815a2c9b6c6c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-21T13:03:38.693 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:38 smithi177 podman[87509]: 2024-03-21 13:03:38.280160964 +0000 UTC m=+0.018544787 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:03:38.693 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:38 smithi177 podman[87509]: 2024-03-21 13:03:38.58201089 +0000 UTC m=+0.320394717 container init 277fc9e56d205e71206198e6a89cd78ce03fb0e306300080e761815a2c9b6c6c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0) 2024-03-21T13:03:38.693 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:38 smithi177 podman[87509]: 2024-03-21 13:03:38.58524071 +0000 UTC m=+0.323624513 container start 277fc9e56d205e71206198e6a89cd78ce03fb0e306300080e761815a2c9b6c6c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T13:03:38.693 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:38 smithi177 podman[87509]: 2024-03-21 13:03:38.691020835 +0000 UTC m=+0.429404655 container attach 277fc9e56d205e71206198e6a89cd78ce03fb0e306300080e761815a2c9b6c6c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True) 2024-03-21T13:03:39.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:38 smithi177 ceph-mon[67966]: osdmap e138: 8 total, 7 up, 8 in 2024-03-21T13:03:39.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:38 smithi177 podman[87509]: 2024-03-21 13:03:38.871635741 +0000 UTC m=+0.610019552 container died 277fc9e56d205e71206198e6a89cd78ce03fb0e306300080e761815a2c9b6c6c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-21T13:03:39.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:38 smithi172 ceph-mon[84635]: osdmap e138: 8 total, 7 up, 8 in 2024-03-21T13:03:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:38 smithi172 ceph-mon[90110]: osdmap e138: 8 total, 7 up, 8 in 2024-03-21T13:03:39.740 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 podman[87535]: 2024-03-21 13:03:39.527543258 +0000 UTC m=+0.650815559 container remove 277fc9e56d205e71206198e6a89cd78ce03fb0e306300080e761815a2c9b6c6c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:03:39.741 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-21T13:03:39.741 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 systemd[1]: Stopped Ceph osd.6 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:03:39.741 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.6.service: Consumed 14.271s CPU time. 2024-03-21T13:03:40.094 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:39 smithi177 ceph-mon[67966]: pgmap v234: 161 pgs: 19 peering, 6 stale+active+clean, 136 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:40.094 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:39 smithi177 ceph-mon[67966]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-21T13:03:40.094 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 systemd[1]: Starting Ceph osd.6 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:03:40.094 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 podman[87639]: 2024-03-21 13:03:39.970624505 +0000 UTC m=+0.111640584 container create d40eb344e4ab5287421cb29333e9e5db52c9581cb33e980cb8297da0aa393a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-21T13:03:40.095 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:39 smithi177 podman[87639]: 2024-03-21 13:03:39.877614413 +0000 UTC m=+0.018630459 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:03:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:39 smithi172 ceph-mon[90110]: pgmap v234: 161 pgs: 19 peering, 6 stale+active+clean, 136 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:39 smithi172 ceph-mon[90110]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-21T13:03:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:39 smithi172 ceph-mon[84635]: pgmap v234: 161 pgs: 19 peering, 6 stale+active+clean, 136 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:39 smithi172 ceph-mon[84635]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-21T13:03:40.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 podman[87639]: 2024-03-21 13:03:40.223438125 +0000 UTC m=+0.364454171 container init d40eb344e4ab5287421cb29333e9e5db52c9581cb33e980cb8297da0aa393a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:03:40.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 podman[87639]: 2024-03-21 13:03:40.227070495 +0000 UTC m=+0.368086535 container start d40eb344e4ab5287421cb29333e9e5db52c9581cb33e980cb8297da0aa393a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:03:40.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 podman[87639]: 2024-03-21 13:03:40.306605282 +0000 UTC m=+0.447621324 container attach d40eb344e4ab5287421cb29333e9e5db52c9581cb33e980cb8297da0aa393a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:03:40.993 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 32/627 objects degraded (5.104%), 11 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T13:03:40.994 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 bash[87639]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate[87653]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-21T13:03:40.995 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:40 smithi177 podman[87639]: 2024-03-21 13:03:40.722944892 +0000 UTC m=+0.863960926 container died d40eb344e4ab5287421cb29333e9e5db52c9581cb33e980cb8297da0aa393a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:03:41.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:40 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 32/627 objects degraded (5.104%), 11 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:40 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 32/627 objects degraded (5.104%), 11 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:41.705 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 podman[87777]: 2024-03-21 13:03:41.360742127 +0000 UTC m=+0.632965387 container remove d40eb344e4ab5287421cb29333e9e5db52c9581cb33e980cb8297da0aa393a37 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:03:41.705 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 podman[87840]: 2024-03-21 13:03:41.477652035 +0000 UTC m=+0.019111766 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:03:41.705 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 podman[87840]: 2024-03-21 13:03:41.595114039 +0000 UTC m=+0.136573771 container create 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T13:03:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:41 smithi177 ceph-mon[67966]: pgmap v235: 161 pgs: 11 active+undersized+degraded, 19 peering, 1 stale+active+clean, 12 active+undersized, 118 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 32/627 objects degraded (5.104%) 2024-03-21T13:03:42.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:41 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:42.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 podman[87840]: 2024-03-21 13:03:41.830996742 +0000 UTC m=+0.372456483 container init 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:03:42.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 podman[87840]: 2024-03-21 13:03:41.834177048 +0000 UTC m=+0.375636788 container start 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-21T13:03:42.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 ceph-osd[87859]: -- 172.21.15.177:0/647780761 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e7a7dd5680 con 0x55e7a7d9f000 2024-03-21T13:03:42.109 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 bash[87840]: 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 2024-03-21T13:03:42.109 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:41 smithi177 systemd[1]: Started Ceph osd.6 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:03:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:41 smithi172 ceph-mon[84635]: pgmap v235: 161 pgs: 11 active+undersized+degraded, 19 peering, 1 stale+active+clean, 12 active+undersized, 118 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 32/627 objects degraded (5.104%) 2024-03-21T13:03:42.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:41 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:41 smithi172 ceph-mon[90110]: pgmap v235: 161 pgs: 11 active+undersized+degraded, 19 peering, 1 stale+active+clean, 12 active+undersized, 118 active+clean; 585 KiB data, 209 MiB used, 715 GiB / 715 GiB avail; 32/627 objects degraded (5.104%) 2024-03-21T13:03:42.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:41 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:42.759 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:03:42.376+0000 7f7cefe617c0 -1 Falling back to public interface 2024-03-21T13:03:43.033 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:43.034 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:43.034 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:42 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:43.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:43.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:42 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:43.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:42 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:43 smithi177 ceph-mon[67966]: pgmap v236: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-21T13:03:44.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:43 smithi177 ceph-mon[67966]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-21T13:03:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:43 smithi172 ceph-mon[84635]: pgmap v236: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-21T13:03:44.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:43 smithi172 ceph-mon[84635]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-21T13:03:44.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:43 smithi172 ceph-mon[90110]: pgmap v236: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-21T13:03:44.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:43 smithi172 ceph-mon[90110]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-03-21T13:03:45.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:03:45.095+0000 7f7cefe617c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-21T13:03:45.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:03:45.123+0000 7f7cefe617c0 -1 osd.6 136 log_to_monitors true 2024-03-21T13:03:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:03:45] "GET /metrics HTTP/1.1" 200 38283 "" "Prometheus/2.43.0" 2024-03-21T13:03:46.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:03:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:03:46.010+0000 7f7ce12b9700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:03:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:45 smithi177 ceph-mon[67966]: pgmap v237: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T13:03:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:45 smithi177 ceph-mon[67966]: from='osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:03:46.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:45 smithi177 ceph-mon[67966]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:03:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-mon[84635]: pgmap v237: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T13:03:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-mon[84635]: from='osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:03:46.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-mon[84635]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:03:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-mon[90110]: pgmap v237: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 79/627 objects degraded (12.600%) 2024-03-21T13:03:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-mon[90110]: from='osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:03:46.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:45 smithi172 ceph-mon[90110]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T13:03:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:47 smithi177 ceph-mon[67966]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T13:03:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:47 smithi177 ceph-mon[67966]: from='osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:47 smithi177 ceph-mon[67966]: osdmap e139: 8 total, 7 up, 8 in 2024-03-21T13:03:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:47 smithi177 ceph-mon[67966]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:47 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[84635]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T13:03:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[84635]: from='osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[84635]: osdmap e139: 8 total, 7 up, 8 in 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[84635]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[90110]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[90110]: from='osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[90110]: osdmap e139: 8 total, 7 up, 8 in 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[90110]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:47.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:47 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: pgmap v239: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037] boot 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 22 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:48.295 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:48 smithi177 ceph-mon[67966]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T13:03:48.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: pgmap v239: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-21T13:03:48.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:03:48.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037] boot 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 22 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[84635]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: pgmap v239: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 229 B/s rd, 0 op/s; 79/627 objects degraded (12.600%) 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: osd.6 [v2:172.21.15.177:6816/3811530037,v1:172.21.15.177:6817/3811530037] boot 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T13:03:48.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:48.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 79/627 objects degraded (12.600%), 22 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:48.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:48 smithi172 ceph-mon[90110]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T13:03:49.292 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:49.293 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:03:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:50 smithi177 ceph-mon[67966]: pgmap v242: 161 pgs: 3 peering, 21 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 70/627 objects degraded (11.164%) 2024-03-21T13:03:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:50 smithi172 ceph-mon[84635]: pgmap v242: 161 pgs: 3 peering, 21 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 70/627 objects degraded (11.164%) 2024-03-21T13:03:50.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:50 smithi172 ceph-mon[90110]: pgmap v242: 161 pgs: 3 peering, 21 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 70/627 objects degraded (11.164%) 2024-03-21T13:03:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: pgmap v243: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: pgmap v244: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: pgmap v245: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:03:52.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: pgmap v243: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: pgmap v244: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: pgmap v245: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: pgmap v243: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: pgmap v244: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: pgmap v245: 161 pgs: 3 peering, 14 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 43/627 objects degraded (6.858%) 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:03:52.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:03:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:03:53.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:03:53.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:03:52.824+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:53.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:03:53.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:53.148 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:53.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:03:53.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:53.151 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:53.152 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:52 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:03:52.835+0000 7f0d06c0d700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:03:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:52 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:03:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:53 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:54.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:54.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:54.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:53 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:03:54.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:53 smithi177 ceph-mon[67966]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-21T13:03:54.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:53 smithi177 ceph-mon[67966]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-21T13:03:54.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:53 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 43/627 objects degraded (6.858%), 14 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:54.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:54.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:54.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:54.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:54.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:03:54.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:03:54.403 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:03:54.404 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[84635]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[84635]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 43/627 objects degraded (6.858%), 14 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:03:54.405 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:03:54.406 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:03:54.406 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:03:54.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:03:54.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[90110]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-21T13:03:54.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[90110]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 691 B/s rd, 0 op/s 2024-03-21T13:03:54.406 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:53 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 43/627 objects degraded (6.858%), 14 pgs degraded (PG_DEGRADED) 2024-03-21T13:03:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:54 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/627 objects degraded (6.858%), 14 pgs degraded) 2024-03-21T13:03:55.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:54 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/627 objects degraded (6.858%), 14 pgs degraded) 2024-03-21T13:03:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:54 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/627 objects degraded (6.858%), 14 pgs degraded) 2024-03-21T13:03:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:03:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:03:55] "GET /metrics HTTP/1.1" 200 38155 "" "Prometheus/2.43.0" 2024-03-21T13:03:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:55 smithi177 ceph-mon[67966]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-21T13:03:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:55 smithi172 ceph-mon[84635]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-21T13:03:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:55 smithi172 ceph-mon[90110]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-21T13:03:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:57 smithi177 ceph-mon[67966]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:57 smithi172 ceph-mon[84635]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:57 smithi172 ceph-mon[90110]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:59.607 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:59.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:03:59 smithi177 ceph-mon[67966]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:59.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:03:59 smithi172 ceph-mon[84635]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:03:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:03:59.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:03:59 smithi172 ceph-mon[90110]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:01.841 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:04:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:02.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:01 smithi172 ceph-mon[84635]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:04:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:01 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:02.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:01 smithi172 ceph-mon[90110]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:04:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:01 smithi177 ceph-mon[67966]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (8m) 111s ago 30m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (8m) 16s ago 30m 78.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 111s ago 29m 68.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (9m) 16s ago 35m 453M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (19m) 111s ago 36m 509M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:04:02.500 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (6m) 111s ago 36m 53.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (7m) 16s ago 35m 48.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (5m) 111s ago 35m 41.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (9m) 111s ago 31m 15.6M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (9m) 16s ago 31m 15.1M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (4m) 111s ago 34m 52.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (3m) 111s ago 34m 51.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (2m) 111s ago 33m 41.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (115s) 111s ago 33m 14.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (79s) 16s ago 33m 53.5M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (57s) 16s ago 32m 43.7M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (21s) 16s ago 32m 14.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:04:02.501 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (32m) 16s ago 32m 62.3M 3248M 17.2.0 e1d6a67b021e 6abfbb59c777 2024-03-21T13:04:02.502 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (8m) 16s ago 31m 75.5M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:04:02.502 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (29m) 111s ago 29m 78.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:04:02.502 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (29m) 16s ago 29m 80.8M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:04:02.912 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:04:02.912 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:04:02.912 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:04:02.912 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 7 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 12 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:04:02.913 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:04:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:02 smithi177 ceph-mon[67966]: from='client.64513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:03.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:02 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3351915020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:03.266 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:02 smithi172 ceph-mon[84635]: from='client.64513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:03.266 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:02 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3351915020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:03.266 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:02 smithi172 ceph-mon[90110]: from='client.64513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:03.266 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:02 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3351915020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "7/8 daemons upgraded", 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading osd daemons", 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:04:03.267 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:04:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:04 smithi177 ceph-mon[67966]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:04 smithi177 ceph-mon[67966]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:04 smithi177 ceph-mon[67966]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:04 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:04 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[84635]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[84635]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[84635]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:04.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[90110]: from='client.54585 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[90110]: from='client.54591 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[90110]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:05.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:05 smithi172 ceph-mon[84635]: from='client.64537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:05.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:05 smithi172 ceph-mon[90110]: from='client.64537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:05.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:05 smithi177 ceph-mon[67966]: from='client.64537 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:04:05] "GET /metrics HTTP/1.1" 200 38301 "" "Prometheus/2.43.0" 2024-03-21T13:04:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:06 smithi177 ceph-mon[67966]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:06 smithi172 ceph-mon[84635]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:06.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:06 smithi172 ceph-mon[90110]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:08 smithi177 ceph-mon[67966]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:08 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:04:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:08 smithi172 ceph-mon[84635]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:08 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:04:08.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:08 smithi172 ceph-mon[90110]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:08.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:08 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:04:09.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:09 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:04:09.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:09 smithi177 ceph-mon[67966]: Upgrade: osd.7 is safe to restart 2024-03-21T13:04:09.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:09 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:04:09.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:09 smithi172 ceph-mon[84635]: Upgrade: osd.7 is safe to restart 2024-03-21T13:04:09.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:09 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-21T13:04:09.400 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:09 smithi172 ceph-mon[90110]: Upgrade: osd.7 is safe to restart 2024-03-21T13:04:10.342 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:10 smithi177 ceph-mon[67966]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:10.342 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:10.342 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T13:04:10.342 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:10.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[84635]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:10.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:10.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T13:04:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[90110]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T13:04:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:11 smithi172 ceph-mon[84635]: Upgrade: Updating osd.7 2024-03-21T13:04:11.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:11 smithi172 ceph-mon[84635]: Deploying daemon osd.7 on smithi177 2024-03-21T13:04:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:11 smithi172 ceph-mon[90110]: Upgrade: Updating osd.7 2024-03-21T13:04:11.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:11 smithi172 ceph-mon[90110]: Deploying daemon osd.7 on smithi177 2024-03-21T13:04:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:11 smithi177 ceph-mon[67966]: Upgrade: Updating osd.7 2024-03-21T13:04:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:11 smithi177 ceph-mon[67966]: Deploying daemon osd.7 on smithi177 2024-03-21T13:04:12.328 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:12 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:12.328 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:12 smithi177 ceph-mon[67966]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:12.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:12 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:12.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:12 smithi172 ceph-mon[84635]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:12.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:12 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:12.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:12 smithi172 ceph-mon[90110]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:12.941 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:12 smithi177 systemd[1]: Stopping Ceph osd.7 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:04:13.064 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:13 smithi172 ceph-mon[84635]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:13 smithi177 ceph-mon[67966]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:13 smithi177 ceph-mon[67966]: osd.7 marked itself down and dead 2024-03-21T13:04:13.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[37914]: 2024-03-21T13:04:12.940+0000 7f0dd6476700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:04:13.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[37914]: 2024-03-21T13:04:12.940+0000 7f0dd6476700 -1 osd.7 141 *** Got signal Terminated *** 2024-03-21T13:04:13.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:12 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[37914]: 2024-03-21T13:04:12.940+0000 7f0dd6476700 -1 osd.7 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:04:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:13 smithi172 ceph-mon[84635]: osd.7 marked itself down and dead 2024-03-21T13:04:13.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:13 smithi172 ceph-mon[90110]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:13.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:13 smithi172 ceph-mon[90110]: osd.7 marked itself down and dead 2024-03-21T13:04:13.658 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:13 smithi177 podman[91939]: 2024-03-21 13:04:13.391649103 +0000 UTC m=+0.571090891 container died 6abfbb59c7774766c59af2e3c23cfc8044f53d028caef459a24c4c21b8412ebe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, version=8, ceph=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-03-21T13:04:14.064 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:13 smithi177 podman[91939]: 2024-03-21 13:04:13.765757522 +0000 UTC m=+0.945199295 container cleanup 6abfbb59c7774766c59af2e3c23cfc8044f53d028caef459a24c4c21b8412ebe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.openshift.expose-services=, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-21T13:04:14.065 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:13 smithi177 bash[91939]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7 2024-03-21T13:04:14.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[91956]: 2024-03-21 13:04:14.067190777 +0000 UTC m=+0.671370215 container remove 6abfbb59c7774766c59af2e3c23cfc8044f53d028caef459a24c4c21b8412ebe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-type=git, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=) 2024-03-21T13:04:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:14 smithi177 ceph-mon[67966]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:04:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:14 smithi177 ceph-mon[67966]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-21T13:04:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:14 smithi177 ceph-mon[67966]: osdmap e142: 8 total, 7 up, 8 in 2024-03-21T13:04:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:14 smithi172 ceph-mon[90110]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:04:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:14 smithi172 ceph-mon[90110]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-21T13:04:14.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:14 smithi172 ceph-mon[90110]: osdmap e142: 8 total, 7 up, 8 in 2024-03-21T13:04:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:14 smithi172 ceph-mon[84635]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T13:04:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:14 smithi172 ceph-mon[84635]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-21T13:04:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:14 smithi172 ceph-mon[84635]: osdmap e142: 8 total, 7 up, 8 in 2024-03-21T13:04:14.762 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[92051]: 2024-03-21 13:04:14.352435014 +0000 UTC m=+0.018081960 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:04:14.762 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[92051]: 2024-03-21 13:04:14.468281653 +0000 UTC m=+0.133928587 container create b5cbf6935cbce0ca0d25f0703773d71319b440d3a8d5e16e5300153fff3db99a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T13:04:14.762 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[92051]: 2024-03-21 13:04:14.679264069 +0000 UTC m=+0.344911011 container init b5cbf6935cbce0ca0d25f0703773d71319b440d3a8d5e16e5300153fff3db99a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-21T13:04:14.762 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[92051]: 2024-03-21 13:04:14.682325276 +0000 UTC m=+0.347972212 container start b5cbf6935cbce0ca0d25f0703773d71319b440d3a8d5e16e5300153fff3db99a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-21T13:04:14.762 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[92051]: 2024-03-21 13:04:14.76252996 +0000 UTC m=+0.428176892 container attach b5cbf6935cbce0ca0d25f0703773d71319b440d3a8d5e16e5300153fff3db99a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-21T13:04:15.077 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:14 smithi177 podman[92051]: 2024-03-21 13:04:14.974676895 +0000 UTC m=+0.640323819 container died b5cbf6935cbce0ca0d25f0703773d71319b440d3a8d5e16e5300153fff3db99a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319) 2024-03-21T13:04:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:15 smithi177 ceph-mon[67966]: osdmap e143: 8 total, 7 up, 8 in 2024-03-21T13:04:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:15 smithi177 ceph-mon[67966]: pgmap v259: 161 pgs: 10 active+undersized+degraded, 17 stale+active+clean, 13 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s; 25/627 objects degraded (3.987%) 2024-03-21T13:04:15.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:04:15] "GET /metrics HTTP/1.1" 200 38301 "" "Prometheus/2.43.0" 2024-03-21T13:04:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:15 smithi172 ceph-mon[84635]: osdmap e143: 8 total, 7 up, 8 in 2024-03-21T13:04:15.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:15 smithi172 ceph-mon[84635]: pgmap v259: 161 pgs: 10 active+undersized+degraded, 17 stale+active+clean, 13 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s; 25/627 objects degraded (3.987%) 2024-03-21T13:04:15.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:15 smithi172 ceph-mon[90110]: osdmap e143: 8 total, 7 up, 8 in 2024-03-21T13:04:15.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:15 smithi172 ceph-mon[90110]: pgmap v259: 161 pgs: 10 active+undersized+degraded, 17 stale+active+clean, 13 active+undersized, 121 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 23 op/s; 25/627 objects degraded (3.987%) 2024-03-21T13:04:15.858 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:15 smithi177 podman[92077]: 2024-03-21 13:04:15.641461298 +0000 UTC m=+0.660550657 container remove b5cbf6935cbce0ca0d25f0703773d71319b440d3a8d5e16e5300153fff3db99a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-21T13:04:15.858 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:15 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-21T13:04:15.858 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:15 smithi177 systemd[1]: Stopped Ceph osd.7 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:04:15.859 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:15 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.7.service: Consumed 17.638s CPU time. 2024-03-21T13:04:16.210 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-mon[67966]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 10 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:16.210 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:15 smithi177 systemd[1]: Starting Ceph osd.7 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:04:16.210 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 podman[92175]: 2024-03-21 13:04:15.995578771 +0000 UTC m=+0.018232031 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:04:16.210 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 podman[92175]: 2024-03-21 13:04:16.101159328 +0000 UTC m=+0.123812582 container create b9b4228b145d9348142b9c40409568b192af417ed171d30cebe314ed0a1fc21e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-21T13:04:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:16 smithi172 ceph-mon[84635]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 10 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:16.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:16 smithi172 ceph-mon[90110]: Health check failed: Degraded data redundancy: 25/627 objects degraded (3.987%), 10 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:16.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 podman[92175]: 2024-03-21 13:04:16.338043292 +0000 UTC m=+0.360696552 container init b9b4228b145d9348142b9c40409568b192af417ed171d30cebe314ed0a1fc21e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-21T13:04:16.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 podman[92175]: 2024-03-21 13:04:16.341496025 +0000 UTC m=+0.364149283 container start b9b4228b145d9348142b9c40409568b192af417ed171d30cebe314ed0a1fc21e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:04:16.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 podman[92175]: 2024-03-21 13:04:16.420696737 +0000 UTC m=+0.443350003 container attach b9b4228b145d9348142b9c40409568b192af417ed171d30cebe314ed0a1fc21e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-21T13:04:17.085 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:04:17.085 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:04:17.085 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 bash[92175]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-21T13:04:17.086 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-21T13:04:17.087 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-21T13:04:17.087 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-21T13:04:17.087 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate[92190]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-21T13:04:17.087 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:16 smithi177 podman[92175]: 2024-03-21 13:04:16.859591635 +0000 UTC m=+0.882244895 container died b9b4228b145d9348142b9c40409568b192af417ed171d30cebe314ed0a1fc21e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-21T13:04:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:17 smithi177 ceph-mon[67966]: pgmap v260: 161 pgs: 25 active+undersized+degraded, 1 stale+active+clean, 39 active+undersized, 96 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 80 op/s; 80/627 objects degraded (12.759%) 2024-03-21T13:04:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:17 smithi172 ceph-mon[84635]: pgmap v260: 161 pgs: 25 active+undersized+degraded, 1 stale+active+clean, 39 active+undersized, 96 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 80 op/s; 80/627 objects degraded (12.759%) 2024-03-21T13:04:17.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:17 smithi172 ceph-mon[90110]: pgmap v260: 161 pgs: 25 active+undersized+degraded, 1 stale+active+clean, 39 active+undersized, 96 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 0 B/s wr, 80 op/s; 80/627 objects degraded (12.759%) 2024-03-21T13:04:17.774 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 podman[92327]: 2024-03-21 13:04:17.458105133 +0000 UTC m=+0.592487090 container remove b9b4228b145d9348142b9c40409568b192af417ed171d30cebe314ed0a1fc21e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-activate, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:04:17.774 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 podman[92394]: 2024-03-21 13:04:17.573657595 +0000 UTC m=+0.018062156 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:04:17.774 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 podman[92394]: 2024-03-21 13:04:17.67567103 +0000 UTC m=+0.120075621 container create 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T13:04:18.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 podman[92394]: 2024-03-21 13:04:17.853278651 +0000 UTC m=+0.297683225 container init 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T13:04:18.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 podman[92394]: 2024-03-21 13:04:17.856492233 +0000 UTC m=+0.300896791 container start 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:04:18.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 ceph-osd[92413]: -- 172.21.15.177:0/2592931441 <== mon.2 v2:172.21.15.177:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f261a67680 con 0x55f261a31000 2024-03-21T13:04:18.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 bash[92394]: 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf 2024-03-21T13:04:18.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:17 smithi177 systemd[1]: Started Ceph osd.7 for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:04:18.791 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:18 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:04:18.399+0000 7f520cec27c0 -1 Falling back to public interface 2024-03-21T13:04:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:19 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:19 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:20 smithi177 ceph-mon[67966]: pgmap v261: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 102 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:20.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:20 smithi172 ceph-mon[84635]: pgmap v261: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 102 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:20.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:20 smithi172 ceph-mon[90110]: pgmap v261: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 0 B/s wr, 102 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:21.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:04:21.104+0000 7f520cec27c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-21T13:04:21.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:21 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:04:21.135+0000 7f520cec27c0 -1 osd.7 141 log_to_monitors true 2024-03-21T13:04:22.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:04:22 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:04:22.067+0000 7f51fe31a700 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:04:22.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:22 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:22.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:22 smithi177 ceph-mon[67966]: pgmap v262: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 144 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:22.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:22 smithi177 ceph-mon[67966]: from='osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:04:22.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:22 smithi177 ceph-mon[67966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:04:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[84635]: pgmap v262: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 144 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[84635]: from='osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:04:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[84635]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:04:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[90110]: pgmap v262: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 144 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[90110]: from='osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:04:22.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:22 smithi172 ceph-mon[90110]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: from='osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: osdmap e144: 8 total, 7 up, 8 in 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:23 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T13:04:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: from='osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:04:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: osdmap e144: 8 total, 7 up, 8 in 2024-03-21T13:04:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:04:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: from='osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: osdmap e144: 8 total, 7 up, 8 in 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi177", "root=default"]}]: dispatch 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:23.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:23 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 82/627 objects degraded (13.078%), 27 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: pgmap v264: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 159 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635] boot 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:24.314 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: pgmap v264: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 159 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:04:24.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635] boot 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: pgmap v264: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 0 B/s wr, 159 op/s; 82/627 objects degraded (13.078%) 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: osd.7 [v2:172.21.15.177:6824/2899926635,v1:172.21.15.177:6825/2899926635] boot 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T13:04:24.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T13:04:24.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:24.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:25.281 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:25 smithi177 ceph-mon[67966]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T13:04:25.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:25 smithi172 ceph-mon[84635]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T13:04:25.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:25 smithi172 ceph-mon[90110]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T13:04:25.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:04:25] "GET /metrics HTTP/1.1" 200 38377 "" "Prometheus/2.43.0" 2024-03-21T13:04:26.330 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:26 smithi177 ceph-mon[67966]: pgmap v267: 161 pgs: 20 active+undersized+degraded, 32 active+undersized, 109 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 68/627 objects degraded (10.845%); 0 B/s, 1 objects/s recovering 2024-03-21T13:04:26.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:26 smithi172 ceph-mon[84635]: pgmap v267: 161 pgs: 20 active+undersized+degraded, 32 active+undersized, 109 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 68/627 objects degraded (10.845%); 0 B/s, 1 objects/s recovering 2024-03-21T13:04:26.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:26 smithi172 ceph-mon[90110]: pgmap v267: 161 pgs: 20 active+undersized+degraded, 32 active+undersized, 109 active+clean; 585 KiB data, 240 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 0 B/s wr, 99 op/s; 68/627 objects degraded (10.845%); 0 B/s, 1 objects/s recovering 2024-03-21T13:04:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: pgmap v268: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 127 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 44 op/s; 50/627 objects degraded (7.974%); 0 B/s, 2 objects/s recovering 2024-03-21T13:04:28.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: pgmap v269: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 127 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%); 0 B/s, 3 objects/s recovering 2024-03-21T13:04:28.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:28.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:28.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:28.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:27 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: pgmap v268: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 127 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 44 op/s; 50/627 objects degraded (7.974%); 0 B/s, 2 objects/s recovering 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: pgmap v269: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 127 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%); 0 B/s, 3 objects/s recovering 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: pgmap v268: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 127 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 44 op/s; 50/627 objects degraded (7.974%); 0 B/s, 2 objects/s recovering 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: pgmap v269: 161 pgs: 13 active+undersized+degraded, 21 active+undersized, 127 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 50/627 objects degraded (7.974%); 0 B/s, 3 objects/s recovering 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:28.148 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:27 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:28.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-21T13:04:28.897 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-21T13:04:28.898 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-21T13:04:28.899 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-21T13:04:28.900 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:04:28.641+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:28.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:28.902 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:28.903 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:28.904 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:28.905 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:28 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: Health check update: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded (PG_DEGRADED) 2024-03-21T13:04:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:29.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-21T13:04:29.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-21T13:04:29.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-21T13:04:29.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-21T13:04:29.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:28 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:30.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:04:30.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:30.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all osd 2024-03-21T13:04:30.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: Upgrade: Setting require_osd_release to 19 squid 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 352 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.114 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:29 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:30.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:30.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:30.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:30.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:30.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:30.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:30.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:30.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all osd 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: Upgrade: Setting require_osd_release to 19 squid 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 352 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all osd 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: Upgrade: Setting require_osd_release to 19 squid 2024-03-21T13:04:30.156 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 352 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.157 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:29 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.852 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mds 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded) 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nfs 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.853 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Finalizing container_image settings 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:04:30.854 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mds 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded) 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nfs 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.855 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Finalizing container_image settings 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:04:30.856 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:04:30.857 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:04:30.857 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:04:30.857 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T13:04:30.857 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:04:30.857 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T13:04:30.857 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.858 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Upgrade: Complete! 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 355 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 0 op/s; 0 B/s, 9 objects/s recovering 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:30.859 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:04:30.860 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Upgrade: Complete! 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:30.861 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 355 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-03-21T13:04:30.862 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 0 op/s; 0 B/s, 9 objects/s recovering 2024-03-21T13:04:30.862 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:30.862 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:30.862 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:30.862 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:30.862 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:30 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mds 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/627 objects degraded (7.974%), 13 pgs degraded) 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nfs 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.109 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Finalizing container_image settings 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:04:31.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:04:31.111 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Upgrade: Complete! 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 355 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-03-21T13:04:31.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 419 B/s rd, 0 op/s; 0 B/s, 9 objects/s recovering 2024-03-21T13:04:31.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:31.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:31.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:31.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:31.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:30 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:31.818 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:04:31.671+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:31.818 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:31.818 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:31.819 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:31.820 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:31.821 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:31.822 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:31.823 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:32.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:31 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:32.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:32.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:32.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:31 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:32 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:33.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:32 smithi177 ceph-mon[67966]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:32 smithi177 ceph-mon[67966]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:32 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:33.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:32 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.146 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:04:33.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:33.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[84635]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[84635]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:33.154 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[90110]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[90110]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 244 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s; 0 B/s, 7 objects/s recovering 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:33.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:32 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:04:33.605+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.897 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.898 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:33.899 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:33.900 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:33.901 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:33 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:34.218 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:34.218 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:34.218 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:34.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:34.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:34.219 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:34.241 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T13:04:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:35.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:35.147 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:35.148 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.149 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:35.150 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:35.151 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:35.152 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:35.153 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.154 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-mon[84635]: from='client.54612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-mon[84635]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-mon[90110]: from='client.54612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:35.155 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-mon[90110]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:35 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:04:35.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:35.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:35 smithi177 ceph-mon[67966]: from='client.54612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:35.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:35 smithi177 ceph-mon[67966]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:04:35.542 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:04:35] "GET /metrics HTTP/1.1" 200 38388 "" "Prometheus/2.43.0" 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (9m) 2m ago 31m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (8m) 14s ago 30m 81.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 2m ago 30m 68.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (10m) 14s ago 35m 453M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (20m) 2m ago 37m 509M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (6m) 2m ago 37m 53.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:04:36.283 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (7m) 14s ago 35m 48.4M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (6m) 2m ago 36m 41.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (9m) 2m ago 31m 15.6M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (9m) 14s ago 31m 15.2M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (4m) 2m ago 35m 52.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (4m) 2m ago 34m 51.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (3m) 2m ago 34m 41.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (2m) 2m ago 34m 14.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (113s) 14s ago 33m 56.5M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (91s) 14s ago 33m 45.7M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (54s) 14s ago 33m 46.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (18s) 14s ago 32m 12.4M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 5e2e973f52fa 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (9m) 14s ago 31m 75.5M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (30m) 2m ago 30m 78.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:04:36.284 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (30m) 14s ago 30m 81.3M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:04:36.982 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-03-21T13:04:37.291 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:37 smithi172 ceph-mon[84635]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-21T13:04:37.292 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:37 smithi172 ceph-mon[90110]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-21T13:04:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:37 smithi177 ceph-mon[67966]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-21T13:04:38.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:38 smithi172 ceph-mon[84635]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:38.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:38 smithi172 ceph-mon[90110]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:38 smithi177 ceph-mon[67966]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:39.347 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:04:39.347 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:39 smithi172 ceph-mon[84635]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:04:39.348 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:39 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:39.348 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:39 smithi172 ceph-mon[90110]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:04:39.348 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:39 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:39 smithi177 ceph-mon[67966]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:04:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:39 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:39.913 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-03-21T13:04:40.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:40 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3626465245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:40.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:40 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3626465245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:40 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3626465245' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:41 smithi172 ceph-mon[90110]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:04:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:41 smithi172 ceph-mon[84635]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:04:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:41 smithi177 ceph-mon[67966]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:04:41.990 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)" 2024-03-21T13:04:42.288 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:42 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:42.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:42 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/4065751472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:42.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:42 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:42.289 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:42 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/4065751472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:42 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:42 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/4065751472' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:42.671 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T13:04:43.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:43 smithi172 ceph-mon[84635]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:04:43.285 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:43 smithi172 ceph-mon[90110]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:04:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:43 smithi177 ceph-mon[67966]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:04:44.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:44 smithi172 ceph-mon[84635]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:44 smithi172 ceph-mon[90110]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:44 smithi177 ceph-mon[67966]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:44.571 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:04:44.571 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:04:44.572 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:04:45.272 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:45 smithi172 ceph-mon[90110]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:45.272 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:45 smithi172 ceph-mon[84635]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:45.297 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T13:04:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:45 smithi177 ceph-mon[67966]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:45.606 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:04:45] "GET /metrics HTTP/1.1" 200 38388 "" "Prometheus/2.43.0" 2024-03-21T13:04:46.246 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:46 smithi172 ceph-mon[84635]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:46.246 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:46 smithi172 ceph-mon[90110]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:46 smithi177 ceph-mon[67966]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:47.325 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:04:47.325 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T13:04:47.325 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T13:04:47.636 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:47 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1389853887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:04:47.636 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:47 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1389853887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:04:47.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:47 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1389853887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:04:47.960 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-03-21T13:04:48.565 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:48 smithi172 ceph-mon[84635]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:48.565 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:48.565 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:48 smithi172 ceph-mon[90110]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:48.565 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:48 smithi177 ceph-mon[67966]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:48.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:04:51.027 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:50 smithi172 ceph-mon[84635]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:51.027 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:50 smithi172 ceph-mon[84635]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:51.027 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:50 smithi172 ceph-mon[90110]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:51.028 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:50 smithi172 ceph-mon[90110]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:51.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:50 smithi177 ceph-mon[67966]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:04:51.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:50 smithi177 ceph-mon[67966]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:52.006 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:52.007 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:52.007 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:52.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:04:52.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-21T13:04:53.246 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:53.246 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:53.246 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:53.246 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:53.247 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:53 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:04:53.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:53 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[84635]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[84635]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[90110]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[90110]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:54.354 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:54 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:54 smithi177 ceph-mon[67966]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-21T13:04:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:54 smithi177 ceph-mon[67966]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:54 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:04:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:54 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:04:54.805 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (9m) 2m ago 31m 26.3M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (9m) 33s ago 31m 81.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 2m ago 30m 68.6M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (10m) 33s ago 35m 453M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (20m) 2m ago 37m 509M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (7m) 2m ago 37m 53.0M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (8m) 33s ago 36m 48.4M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (6m) 2m ago 36m 41.6M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (10m) 2m ago 32m 15.6M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (10m) 33s ago 32m 15.2M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:04:55.465 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (5m) 2m ago 35m 52.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (4m) 2m ago 35m 51.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (3m) 2m ago 34m 41.5M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (2m) 2m ago 34m 14.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (2m) 33s ago 34m 56.5M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (110s) 33s ago 33m 45.7M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (73s) 33s ago 33m 46.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (37s) 33s ago 32m 12.4M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 5e2e973f52fa 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (9m) 33s ago 32m 75.5M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (30m) 2m ago 30m 78.0M - 17.2.0 e1d6a67b021e ddb1bc9cb0f1 2024-03-21T13:04:55.466 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (30m) 33s ago 30m 81.3M - 17.2.0 e1d6a67b021e 09f2d41dac97 2024-03-21T13:04:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:04:55] "GET /metrics HTTP/1.1" 200 38381 "" "Prometheus/2.43.0" 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:55.880 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 8 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 13 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:04:55.881 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:04:56.013 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:04:55.891+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:56.013 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:56.013 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:56.013 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:56.013 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:56.014 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:56.015 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:04:56.016 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:56.017 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:56.018 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:56.018 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:04:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "", 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 image", 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:04:56.234 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:04:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:56 smithi177 ceph-mon[67966]: from='client.54654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:56 smithi177 ceph-mon[67966]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:56.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:56 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/437145134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:56 smithi172 ceph-mon[84635]: from='client.54654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:56 smithi172 ceph-mon[84635]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:56.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:56 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/437145134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:56 smithi172 ceph-mon[90110]: from='client.54654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:56 smithi172 ceph-mon[90110]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:56.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:56 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/437145134' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[84635]: from='client.64594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[84635]: from='client.54666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:57.119 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:57.120 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:57.121 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:57.122 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[90110]: from='client.64594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[90110]: from='client.54666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:57.123 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:57.124 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:57.125 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:57.126 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[90110]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.127 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:57 smithi172 ceph-mon[84635]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:57 smithi177 ceph-mon[67966]: from='client.64594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:57 smithi177 ceph-mon[67966]: from='client.54666 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:57 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:57.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:04:57.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:04:57.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:04:57.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:57 smithi177 ceph-mon[67966]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:04:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:58 smithi177 ceph-mon[67966]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:58 smithi177 ceph-mon[67966]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:58 smithi172 ceph-mon[84635]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:58 smithi172 ceph-mon[84635]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:58 smithi172 ceph-mon[90110]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:04:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:58 smithi172 ceph-mon[90110]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:04:59.332 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.332 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:04:59.332 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mgr 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mon 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all crash 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all osd 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.333 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mds 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mgr 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mon 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.334 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all crash 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all osd 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mds 2024-03-21T13:04:59.335 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:04:59 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mgr 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mon 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all osd 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mds 2024-03-21T13:04:59.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:04:59 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:00.339 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[84635]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[90110]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:05:00.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:00 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:00 smithi177 ceph-mon[67966]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:00 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:00 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.aecdpn", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:05:00.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:00 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:01.354 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:01 smithi172 ceph-mon[84635]: Upgrade: Updating rgw.foo.smithi172.aecdpn (1/2) 2024-03-21T13:05:01.355 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:01 smithi172 ceph-mon[84635]: Deploying daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T13:05:01.355 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:01 smithi172 ceph-mon[90110]: Upgrade: Updating rgw.foo.smithi172.aecdpn (1/2) 2024-03-21T13:05:01.355 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:01 smithi172 ceph-mon[90110]: Deploying daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T13:05:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:01 smithi177 ceph-mon[67966]: Upgrade: Updating rgw.foo.smithi172.aecdpn (1/2) 2024-03-21T13:05:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:01 smithi177 ceph-mon[67966]: Deploying daemon rgw.foo.smithi172.aecdpn on smithi172 2024-03-21T13:05:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:02 smithi177 ceph-mon[67966]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-21T13:05:02.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:02 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:02.387 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:02 smithi172 ceph-mon[84635]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-21T13:05:02.387 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:02 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:02.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:02 smithi172 ceph-mon[90110]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 559 B/s rd, 0 op/s 2024-03-21T13:05:02.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:02 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:03.395 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:03.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:04 smithi177 ceph-mon[67966]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:05:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:04 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:04.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:04 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:04 smithi172 ceph-mon[84635]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:05:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:04 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:04.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:04 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:04 smithi172 ceph-mon[90110]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T13:05:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:04.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:04 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:05.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:05:05] "GET /metrics HTTP/1.1" 200 38379 "" "Prometheus/2.43.0" 2024-03-21T13:05:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:06 smithi177 ceph-mon[67966]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-21T13:05:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:05:06.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[84635]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-21T13:05:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:05:06.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[90110]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-03-21T13:05:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi177.magnlc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T13:05:06.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:07.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:07 smithi177 ceph-mon[67966]: Upgrade: Updating rgw.foo.smithi177.magnlc (2/2) 2024-03-21T13:05:07.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:07 smithi177 ceph-mon[67966]: Deploying daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T13:05:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:07 smithi172 ceph-mon[84635]: Upgrade: Updating rgw.foo.smithi177.magnlc (2/2) 2024-03-21T13:05:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:07 smithi172 ceph-mon[84635]: Deploying daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T13:05:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:07 smithi172 ceph-mon[90110]: Upgrade: Updating rgw.foo.smithi177.magnlc (2/2) 2024-03-21T13:05:07.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:07 smithi172 ceph-mon[90110]: Deploying daemon rgw.foo.smithi177.magnlc on smithi177 2024-03-21T13:05:08.125 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:08 smithi177 ceph-mon[67966]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 56 op/s 2024-03-21T13:05:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:08 smithi172 ceph-mon[84635]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 56 op/s 2024-03-21T13:05:08.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:08 smithi172 ceph-mon[90110]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 56 op/s 2024-03-21T13:05:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:09 smithi177 ceph-mon[67966]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-21T13:05:09.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:09 smithi172 ceph-mon[84635]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-21T13:05:09.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:09 smithi172 ceph-mon[90110]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 85 B/s wr, 84 op/s 2024-03-21T13:05:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:10.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:10.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:10.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:10.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:10.796 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:11.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:11 smithi177 ceph-mon[67966]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 97 op/s 2024-03-21T13:05:11.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:11 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:11 smithi172 ceph-mon[84635]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 97 op/s 2024-03-21T13:05:11.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:11 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:11 smithi172 ceph-mon[90110]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 97 op/s 2024-03-21T13:05:11.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:11 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[84635]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 170 B/s wr, 156 op/s 2024-03-21T13:05:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[90110]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 170 B/s wr, 156 op/s 2024-03-21T13:05:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:13 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:13 smithi177 ceph-mon[67966]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s rd, 170 B/s wr, 156 op/s 2024-03-21T13:05:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:13 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:15 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.608 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:15 smithi177 ceph-mon[67966]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-21T13:05:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:05:15] "GET /metrics HTTP/1.1" 200 38379 "" "Prometheus/2.43.0" 2024-03-21T13:05:15.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.646 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.647 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[90110]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-21T13:05:15.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:15.647 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:15 smithi172 ceph-mon[84635]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 109 KiB/s rd, 170 B/s wr, 168 op/s 2024-03-21T13:05:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:17 smithi172 ceph-mon[84635]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 170 B/s wr, 144 op/s 2024-03-21T13:05:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:17 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:17 smithi172 ceph-mon[90110]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 170 B/s wr, 144 op/s 2024-03-21T13:05:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:17 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:17 smithi177 ceph-mon[67966]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 170 B/s wr, 144 op/s 2024-03-21T13:05:18.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:17 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:18.993 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 183 B/s wr, 121 op/s 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 111 B/s wr, 110 op/s 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:18.994 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:18.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 183 B/s wr, 121 op/s 2024-03-21T13:05:18.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 111 B/s wr, 110 op/s 2024-03-21T13:05:18.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:18.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:18.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:18.995 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 183 B/s wr, 121 op/s 2024-03-21T13:05:19.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 111 B/s wr, 110 op/s 2024-03-21T13:05:19.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:19.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:19.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:19.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:20.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:19 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:20.011 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:19 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:20.011 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:19 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:20.011 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:19 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:19 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:20.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:19 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:05:20.103+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:20.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:20.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:20.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:20.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:20.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:20.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:05:20.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:20.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:20.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:20.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:20.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:20.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:20.403 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:20 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:21.164 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 111 B/s wr, 94 op/s 2024-03-21T13:05:21.164 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:21.164 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.164 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.164 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.164 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.aecdpn"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.aecdpn"}]': finished 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi177.magnlc"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi177.magnlc"}]': finished 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:05:21.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T13:05:21.167 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 111 B/s wr, 94 op/s 2024-03-21T13:05:21.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.aecdpn"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.aecdpn"}]': finished 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi177.magnlc"}]: dispatch 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi177.magnlc"}]': finished 2024-03-21T13:05:21.169 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.170 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.171 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:05:21.172 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:05:21.173 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.174 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:21.175 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:21 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:21.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 111 B/s wr, 94 op/s 2024-03-21T13:05:21.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:21.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.255 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.aecdpn"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.aecdpn"}]': finished 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi177.magnlc"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi177.magnlc"}]': finished 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.256 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.257 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-21T13:05:21.258 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.259 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:21.260 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:21 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.108 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.109 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:05:22.110 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.111 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rgw 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:05:22.112 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nfs 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Finalizing container_image settings 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Upgrade: Complete! 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Checking dashboard <-> RGW credentials 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:22.113 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:22 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:22.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.394 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.395 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:22.396 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.397 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all rgw 2024-03-21T13:05:22.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nfs 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Finalizing container_image settings 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Upgrade: Complete! 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Checking dashboard <-> RGW credentials 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.399 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all rgw 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nfs 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all nvmeof 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Finalizing container_image settings 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Upgrade: Complete! 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 20 op/s 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 951 B/s rd, 0 op/s 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Checking dashboard <-> RGW credentials 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:22.400 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:22.401 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:05:22.538+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:22.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:22.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:22.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:22.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:22 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:23.274 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:23.274 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:23.275 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:23 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:23.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:23 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:24.277 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4 op/s 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:24.278 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:24.279 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.280 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.281 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4 op/s 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.282 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:24.283 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:24.284 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.285 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:24.286 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:24.287 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 4 op/s 2024-03-21T13:05:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:24.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:24.363 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:24 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:05:24.452+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.648 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:24.649 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:24.650 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:24.651 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:24 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:05:25] "GET /metrics HTTP/1.1" 200 38420 "" "Prometheus/2.43.0" 2024-03-21T13:05:26.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:26 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:26.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:05:26.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:26.363 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:26.364 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.365 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.366 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:26 smithi177 ceph-mon[67966]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:26 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:26.397 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:26.398 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:26.399 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:05:26.400 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:26.401 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:26.402 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:26.403 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:26 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:26.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:26.403 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:26.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:26.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:26.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.404 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:26.405 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.406 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:26.407 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:26.408 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:26.409 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:26 smithi172 ceph-mon[84635]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:26.410 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:26.411 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:26.411 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:26.411 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:26.411 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:26.411 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:26 smithi172 ceph-mon[90110]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.3 KiB/s rd, 5 op/s 2024-03-21T13:05:27.192 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T13:05:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:28 smithi172 ceph-mon[90110]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 5 op/s 2024-03-21T13:05:28.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:28 smithi172 ceph-mon[90110]: from='client.54762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:28.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:28 smithi172 ceph-mon[84635]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 5 op/s 2024-03-21T13:05:28.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:28 smithi172 ceph-mon[84635]: from='client.54762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:28.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:28 smithi177 ceph-mon[67966]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 5 op/s 2024-03-21T13:05:28.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:28 smithi177 ceph-mon[67966]: from='client.54762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (10m) 15s ago 32m 26.4M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (9m) 15s ago 31m 77.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 15s ago 31m 68.8M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (11m) 15s ago 36m 454M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (21m) 15s ago 38m 515M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (7m) 15s ago 38m 61.3M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (8m) 15s ago 36m 50.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:05:29.158 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (7m) 15s ago 36m 47.5M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (10m) 15s ago 32m 15.8M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (10m) 15s ago 32m 15.3M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (5m) 15s ago 36m 61.2M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (5m) 15s ago 35m 59.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (4m) 15s ago 35m 48.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (3m) 15s ago 35m 52.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (2m) 15s ago 34m 60.2M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (2m) 15s ago 34m 49.4M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (107s) 15s ago 33m 49.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (71s) 15s ago 33m 47.9M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 5e2e973f52fa 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (10m) 15s ago 32m 76.7M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (25s) 15s ago 31m 77.4M - 19.0.0-2267-g960fbbe4 19e12293512f d404df3853a8 2024-03-21T13:05:29.159 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (20s) 15s ago 31m 74.9M - 19.0.0-2267-g960fbbe4 19e12293512f d69073373deb 2024-03-21T13:05:29.789 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-03-21T13:05:30.327 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:30 smithi172 ceph-mon[84635]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 4 op/s 2024-03-21T13:05:30.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:30 smithi172 ceph-mon[90110]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 4 op/s 2024-03-21T13:05:30.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:30 smithi177 ceph-mon[67966]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 4 op/s 2024-03-21T13:05:31.224 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:31 smithi172 ceph-mon[84635]: from='client.64699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:31.224 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:31 smithi172 ceph-mon[90110]: from='client.64699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:31 smithi177 ceph-mon[67966]: from='client.64699 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:31.919 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:05:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:32 smithi177 ceph-mon[67966]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-03-21T13:05:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:32 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:32.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:32 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/178589104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:32.365 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:32 smithi172 ceph-mon[84635]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-03-21T13:05:32.365 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:32 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:32.365 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:32 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/178589104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:32.365 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:32 smithi172 ceph-mon[90110]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-03-21T13:05:32.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:32 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:32.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:32 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/178589104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:32.698 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-03-21T13:05:34.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:34 smithi172 ceph-mon[84635]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:05:34.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:34 smithi172 ceph-mon[90110]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:05:34.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:34 smithi177 ceph-mon[67966]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:05:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:34.703 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)" 2024-03-21T13:05:35.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:35 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1964082316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:35 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1964082316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:35 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1964082316' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:35.497 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T13:05:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:05:35] "GET /metrics HTTP/1.1" 200 38418 "" "Prometheus/2.43.0" 2024-03-21T13:05:36.110 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:36 smithi172 ceph-mon[84635]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:36.110 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:36 smithi172 ceph-mon[90110]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:36.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:36 smithi177 ceph-mon[67966]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:37.662 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:05:37.662 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": null, 2024-03-21T13:05:37.662 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": false, 2024-03-21T13:05:37.662 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "", 2024-03-21T13:05:37.662 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [], 2024-03-21T13:05:37.663 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": null, 2024-03-21T13:05:37.663 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "", 2024-03-21T13:05:37.663 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:05:37.663 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:05:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:38 smithi177 ceph-mon[67966]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:05:38.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:38 smithi172 ceph-mon[84635]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:05:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:38 smithi172 ceph-mon[90110]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T13:05:38.514 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T13:05:39.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:39 smithi172 ceph-mon[84635]: from='client.64717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:39.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:39 smithi172 ceph-mon[90110]: from='client.64717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:39 smithi177 ceph-mon[67966]: from='client.64717 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:39.858 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:05:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=cleanup t=2024-03-21T13:05:39.491985935Z level=info msg="Completed cleanup jobs" duration=2.241015ms 2024-03-21T13:05:40.266 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:40 smithi172 ceph-mon[84635]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:40.267 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:40 smithi172 ceph-mon[90110]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:40.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:40 smithi177 ceph-mon[67966]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:40.764 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:05:40.764 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T13:05:40.764 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T13:05:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:41 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1308419916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:05:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:41 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1308419916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:05:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:41 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1308419916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:05:41.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-21T13:05:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:42 smithi177 ceph-mon[67966]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:42.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:42 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:42 smithi172 ceph-mon[84635]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:42.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:42 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:42.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:42 smithi172 ceph-mon[90110]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:42.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:42 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:43.503 INFO:teuthology.orchestra.run.smithi172.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:05:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:44.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:44.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:44 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:44.171 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T13:05:44.182 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi172.front.sepia.ceph.com 2024-03-21T13:05:44.182 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-21T13:05:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:05:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:05:44.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:44.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:44.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:44 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[84635]: from='client.54798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[84635]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[84635]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[84635]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:45.238 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[90110]: from='client.54798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:45.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[90110]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:45.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[90110]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-03-21T13:05:45.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[90110]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:45.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:45.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:45 smithi177 ceph-mon[67966]: from='client.54798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:45 smithi177 ceph-mon[67966]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:45 smithi177 ceph-mon[67966]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-03-21T13:05:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:45 smithi177 ceph-mon[67966]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:45 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:05:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:45 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:45.600 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:05:45] "GET /metrics HTTP/1.1" 200 38418 "" "Prometheus/2.43.0" 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:05:46.118+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:46.397 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:46.398 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:46.399 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return future.result(timeout) 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return self.__get_result() 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: raise self._exception 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Creating ceph-iscsi config... 2024-03-21T13:05:46.400 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: Traceback (most recent call last): 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: return _run_code(code, main_globals, None, 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: exec(code, run_globals) 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:46.401 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:46.402 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:46 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:47.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:47 smithi172 ceph-mon[84635]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-03-21T13:05:47.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:47 smithi172 ceph-mon[90110]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-03-21T13:05:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:47 smithi177 ceph-mon[67966]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 571 B/s rd, 0 op/s 2024-03-21T13:05:47.392 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:05:48.088 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:05:48.088 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (10m) 34s ago 32m 26.4M - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (10m) 34s ago 32m 77.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 unknown 34s ago 31m 68.8M - 3.5 e1d6a67b021e ec8767e5fef8 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (11m) 34s ago 36m 454M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (21m) 34s ago 38m 515M - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (8m) 34s ago 38m 61.3M 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (9m) 34s ago 37m 50.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (7m) 34s ago 37m 47.5M 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (11m) 34s ago 33m 15.8M - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (11m) 34s ago 32m 15.3M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (6m) 34s ago 36m 61.2M 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:05:48.089 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (5m) 34s ago 36m 59.1M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (4m) 34s ago 35m 48.7M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (3m) 34s ago 35m 52.9M 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (3m) 34s ago 34m 60.2M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (2m) 34s ago 34m 49.4M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (2m) 34s ago 34m 49.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (90s) 34s ago 33m 47.9M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 5e2e973f52fa 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (10m) 34s ago 33m 76.7M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (44s) 34s ago 31m 77.4M - 19.0.0-2267-g960fbbe4 19e12293512f d404df3853a8 2024-03-21T13:05:48.090 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (39s) 34s ago 31m 74.9M - 19.0.0-2267-g960fbbe4 19e12293512f d69073373deb 2024-03-21T13:05:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:48 smithi177 ceph-mon[67966]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:48.359 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return future.result(timeout) 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: return self.__get_result() 2024-03-21T13:05:48.360 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: raise self._exception 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:48.361 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: exec(code, run_globals) 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:48 smithi177 ceph-mon[67966]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:48.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:48 smithi172 ceph-mon[84635]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.386 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.387 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: raise self._exception 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:48.388 INFO:journalctl@ceph.mon.a.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:48.389 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:48 smithi172 ceph-mon[84635]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:48 smithi172 ceph-mon[90110]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:48.390 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:48.391 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2403, in _daemon_action 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 749, in wait_async 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: return future.result(timeout) 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-21T13:05:48.392 INFO:journalctl@ceph.mon.c.smithi172.stdout: return self.__get_result() 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: raise self._exception 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Creating ceph-iscsi config... 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: Traceback (most recent call last): 2024-03-21T13:05:48.393 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1121, in deploy_daemon 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout: See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:05:48.394 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:48 smithi172 ceph-mon[90110]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:960fbbe4645c2becc8084669be194c95183c7753 2024-03-21T13:05:48.395 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 8 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:05:48.509 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:05:48.510 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:05:48.510 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 15 2024-03-21T13:05:48.510 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:05:48.510 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw", 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr", 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "osd", 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "mon" 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "15/21 daemons upgraded", 2024-03-21T13:05:48.864 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Currently upgrading iscsi daemons", 2024-03-21T13:05:48.865 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": false 2024-03-21T13:05:48.865 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:05:49.293 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-21T13:05:49.293 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T13:05:49.293 INFO:teuthology.orchestra.run.smithi172.stdout: daemon iscsi.foo.smithi172.bkzkmk on smithi172 is in unknown state 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='client.64735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='client.64741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='client.64747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.337 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1207415933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.338 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='client.64735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='client.64741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='client.64747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.339 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1207415933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.340 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='client.64735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='client.64741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='client.64747 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1207415933' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:05:49.360 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mgr 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mon 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all crash 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all osd 2024-03-21T13:05:50.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all mds 2024-03-21T13:05:50.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rgw 2024-03-21T13:05:50.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:05:50.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T13:05:50.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:05:50.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: from='client.64759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:50.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:50 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2052744996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mgr 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mon 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all crash 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all osd 2024-03-21T13:05:50.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all mds 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all rgw 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: from='client.64759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2052744996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Target is version 19.0.0-2267-g960fbbe4 (squid) 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, digests ['quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93'] 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mgr 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mon 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all crash 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all osd 2024-03-21T13:05:50.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all mds 2024-03-21T13:05:50.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all rgw 2024-03-21T13:05:50.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all rbd-mirror 2024-03-21T13:05:50.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-21T13:05:50.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: Upgrade: Setting container_image for all ceph-exporter 2024-03-21T13:05:50.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: from='client.64759 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:05:50.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:50 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2052744996' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:05:51.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: Upgrade: Updating iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: Deploying daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: Upgrade: Updating iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:51.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: Deploying daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:51.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:51 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: Upgrade: Updating iscsi.foo.smithi172.bkzkmk 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: Deploying daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:05:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:51 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:05:52.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:52 smithi172 ceph-mon[84635]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:52.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:52 smithi172 ceph-mon[90110]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:52 smithi177 ceph-mon[67966]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T13:05:54.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:54 smithi177 ceph-mon[67966]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:05:54.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:54 smithi172 ceph-mon[84635]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:05:54.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:54 smithi172 ceph-mon[90110]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:05:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:05:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:05:55] "GET /metrics HTTP/1.1" 200 38421 "" "Prometheus/2.43.0" 2024-03-21T13:05:56.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:56 smithi177 ceph-mon[67966]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:56.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:56 smithi172 ceph-mon[84635]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:56.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:56 smithi172 ceph-mon[90110]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:05:58.832 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:05:58 smithi172 ceph-mon[84635]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:05:58.832 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:05:58 smithi172 ceph-mon[90110]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:05:58.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:05:58 smithi177 ceph-mon[67966]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:00.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:00 smithi177 ceph-mon[67966]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:00.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:00 smithi172 ceph-mon[84635]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:00.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:00 smithi172 ceph-mon[90110]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:01.660 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:01 smithi172 ceph-mon[84635]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:01.660 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:01 smithi172 ceph-mon[90110]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:01.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:01 smithi177 ceph-mon[67966]: from='client.35012 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:02.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:02 smithi177 ceph-mon[67966]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:02.896 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:02 smithi172 ceph-mon[84635]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:02 smithi172 ceph-mon[90110]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:03.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:06:03.139+0000 7f0cd73a5700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. 2024-03-21T13:06:03.745 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:03.746 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:06:03.747 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi172.bkzkmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T13:06:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:06:04.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:04 smithi177 ceph-mon[67966]: Health check failed: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-21T13:06:04.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. 2024-03-21T13:06:04.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-21T13:06:04.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:06:04.885 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[90110]: Health check failed: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 880 B/s rd, 0 op/s 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: Reconfiguring iscsi.foo.smithi172.bkzkmk (dependencies changed)... 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: Reconfiguring daemon iscsi.foo.smithi172.bkzkmk on smithi172 2024-03-21T13:06:04.886 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:04 smithi172 ceph-mon[84635]: Health check failed: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-21T13:06:05.539 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:06:05] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-03-21T13:06:07.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 670 B/s rd, 0 op/s 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/985695762' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2645381451' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/389602532"}]: dispatch 2024-03-21T13:06:07.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 670 B/s rd, 0 op/s 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/985695762' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T13:06:07.148 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:06 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2645381451' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/389602532"}]: dispatch 2024-03-21T13:06:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 670 B/s rd, 0 op/s 2024-03-21T13:06:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T13:06:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-21T13:06:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T13:06:07.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-21T13:06:07.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:06:07.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/985695762' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T13:06:07.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:06 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2645381451' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/389602532"}]: dispatch 2024-03-21T13:06:07.904 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2645381451' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/389602532"}]': finished 2024-03-21T13:06:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:07 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2645381451' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/389602532"}]': finished 2024-03-21T13:06:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:07 smithi177 ceph-mon[67966]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T13:06:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:07 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1875570753' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]: dispatch 2024-03-21T13:06:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:07 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]: dispatch 2024-03-21T13:06:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[84635]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T13:06:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1875570753' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]: dispatch 2024-03-21T13:06:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]: dispatch 2024-03-21T13:06:08.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2645381451' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/389602532"}]': finished 2024-03-21T13:06:08.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[90110]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T13:06:08.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1875570753' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]: dispatch 2024-03-21T13:06:08.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:07 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]: dispatch 2024-03-21T13:06:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[84635]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 545 B/s rd, 0 op/s 2024-03-21T13:06:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]': finished 2024-03-21T13:06:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[84635]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T13:06:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/672505004' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]: dispatch 2024-03-21T13:06:09.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]: dispatch 2024-03-21T13:06:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[90110]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 545 B/s rd, 0 op/s 2024-03-21T13:06:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]': finished 2024-03-21T13:06:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[90110]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T13:06:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/672505004' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]: dispatch 2024-03-21T13:06:09.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:08 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]: dispatch 2024-03-21T13:06:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:08 smithi177 ceph-mon[67966]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 545 B/s rd, 0 op/s 2024-03-21T13:06:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:08 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/3851548894"}]': finished 2024-03-21T13:06:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:08 smithi177 ceph-mon[67966]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T13:06:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:08 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/672505004' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]: dispatch 2024-03-21T13:06:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:08 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]: dispatch 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]': finished 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3942112171' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]: dispatch 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]: dispatch 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:09 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]': finished 2024-03-21T13:06:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T13:06:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3942112171' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]: dispatch 2024-03-21T13:06:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]: dispatch 2024-03-21T13:06:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/2126228407"}]': finished 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3942112171' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]: dispatch 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]: dispatch 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:06:10.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:06:10.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:09 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:06:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-21T13:06:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T13:06:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]': finished 2024-03-21T13:06:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T13:06:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3439038807' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]: dispatch 2024-03-21T13:06:11.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:10 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]: dispatch 2024-03-21T13:06:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:06:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-21T13:06:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T13:06:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]': finished 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3439038807' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]: dispatch 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]: dispatch 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2764915619"}]': finished 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3439038807' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]: dispatch 2024-03-21T13:06:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:10 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]: dispatch 2024-03-21T13:06:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:11 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]': finished 2024-03-21T13:06:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:11 smithi177 ceph-mon[67966]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T13:06:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:11 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3804295638' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1775920990"}]: dispatch 2024-03-21T13:06:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:11 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]': finished 2024-03-21T13:06:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:11 smithi172 ceph-mon[90110]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T13:06:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:11 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3804295638' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1775920990"}]: dispatch 2024-03-21T13:06:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:11 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2817004094"}]': finished 2024-03-21T13:06:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:11 smithi172 ceph-mon[84635]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T13:06:12.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:11 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3804295638' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1775920990"}]: dispatch 2024-03-21T13:06:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:12 smithi177 ceph-mon[67966]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:12 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3804295638' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1775920990"}]': finished 2024-03-21T13:06:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:12 smithi177 ceph-mon[67966]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T13:06:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:12 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/675721025' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]: dispatch 2024-03-21T13:06:13.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:12 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]: dispatch 2024-03-21T13:06:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[84635]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3804295638' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1775920990"}]': finished 2024-03-21T13:06:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[84635]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T13:06:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/675721025' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]: dispatch 2024-03-21T13:06:13.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]: dispatch 2024-03-21T13:06:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[90110]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3804295638' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1775920990"}]': finished 2024-03-21T13:06:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[90110]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T13:06:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/675721025' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]: dispatch 2024-03-21T13:06:13.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:12 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]: dispatch 2024-03-21T13:06:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:13 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]': finished 2024-03-21T13:06:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:13 smithi177 ceph-mon[67966]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T13:06:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:13 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/4293183395' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]: dispatch 2024-03-21T13:06:14.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:13 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]: dispatch 2024-03-21T13:06:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]': finished 2024-03-21T13:06:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[84635]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T13:06:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/4293183395' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]: dispatch 2024-03-21T13:06:14.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]: dispatch 2024-03-21T13:06:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/2417852765"}]': finished 2024-03-21T13:06:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[90110]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T13:06:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/4293183395' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]: dispatch 2024-03-21T13:06:14.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:13 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]: dispatch 2024-03-21T13:06:15.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[84635]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:15.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]': finished 2024-03-21T13:06:15.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[84635]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T13:06:15.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1867666942' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]: dispatch 2024-03-21T13:06:15.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]: dispatch 2024-03-21T13:06:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[90110]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]': finished 2024-03-21T13:06:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[90110]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T13:06:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1867666942' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]: dispatch 2024-03-21T13:06:15.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:14 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]: dispatch 2024-03-21T13:06:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:14 smithi177 ceph-mon[67966]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:14 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1974762506"}]': finished 2024-03-21T13:06:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:14 smithi177 ceph-mon[67966]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T13:06:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:14 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1867666942' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]: dispatch 2024-03-21T13:06:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:14 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]: dispatch 2024-03-21T13:06:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:06:15] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-03-21T13:06:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:15 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]': finished 2024-03-21T13:06:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:15 smithi177 ceph-mon[67966]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T13:06:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:15 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1302686207' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]: dispatch 2024-03-21T13:06:16.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:15 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]: dispatch 2024-03-21T13:06:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]': finished 2024-03-21T13:06:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[84635]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T13:06:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1302686207' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]: dispatch 2024-03-21T13:06:16.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]: dispatch 2024-03-21T13:06:16.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:0/1147503507"}]': finished 2024-03-21T13:06:16.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[90110]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T13:06:16.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1302686207' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]: dispatch 2024-03-21T13:06:16.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:15 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]: dispatch 2024-03-21T13:06:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:16 smithi177 ceph-mon[67966]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:06:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:16 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]': finished 2024-03-21T13:06:17.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:16 smithi177 ceph-mon[67966]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T13:06:17.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:16 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:17.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:16 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1915213095' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]: dispatch 2024-03-21T13:06:17.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:16 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]: dispatch 2024-03-21T13:06:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[84635]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:06:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]': finished 2024-03-21T13:06:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[84635]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T13:06:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:17.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1915213095' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]: dispatch 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]: dispatch 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[90110]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/415955282"}]': finished 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[90110]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1915213095' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]: dispatch 2024-03-21T13:06:17.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:16 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]: dispatch 2024-03-21T13:06:18.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:17 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]': finished 2024-03-21T13:06:18.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:17 smithi177 ceph-mon[67966]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T13:06:18.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:17 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1205399761' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]: dispatch 2024-03-21T13:06:18.362 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:17 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]: dispatch 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]': finished 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[84635]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1205399761' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]: dispatch 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]: dispatch 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6800/2126228407"}]': finished 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[90110]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1205399761' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]: dispatch 2024-03-21T13:06:18.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:17 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]: dispatch 2024-03-21T13:06:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:18 smithi177 ceph-mon[67966]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:18 smithi177 ceph-mon[67966]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]': finished 2024-03-21T13:06:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:18 smithi177 ceph-mon[67966]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T13:06:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[84635]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[84635]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]': finished 2024-03-21T13:06:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[84635]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T13:06:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[90110]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[90110]: from='client.? ' entity='client.iscsi.foo.smithi172.bkzkmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.172:6801/415955282"}]': finished 2024-03-21T13:06:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[90110]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T13:06:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:06:19.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:19.650 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:06:19.980 INFO:teuthology.orchestra.run.smithi172.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed." 2024-03-21T13:06:20.843 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T13:06:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:20 smithi172 ceph-mon[84635]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:20 smithi172 ceph-mon[84635]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:06:21.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:20 smithi172 ceph-mon[84635]: from='client.64873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:20 smithi172 ceph-mon[90110]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:20 smithi172 ceph-mon[90110]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:06:21.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:20 smithi172 ceph-mon[90110]: from='client.64873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:20 smithi177 ceph-mon[67966]: from='client.64867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:20 smithi177 ceph-mon[67966]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T13:06:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:20 smithi177 ceph-mon[67966]: from='client.64873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:22.951 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:06:22.951 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (11m) 13s ago 32m - - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:06:22.951 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (10m) 69s ago 32m 77.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:06:22.951 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (17s) 13s ago 32m - - 3.7 19e12293512f e7ee6631eaed 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (11m) 69s ago 37m 454M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (22m) 13s ago 38m - - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (8m) 13s ago 39m - 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (9m) 69s ago 37m 50.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (8m) 13s ago 37m - 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (11m) 13s ago 33m - - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (11m) 69s ago 33m 15.3M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (6m) 13s ago 37m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (6m) 13s ago 36m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (5m) 13s ago 36m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (4m) 13s ago 35m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (3m) 69s ago 35m 60.2M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (3m) 69s ago 35m 49.4M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:06:22.952 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (2m) 69s ago 34m 49.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:06:22.953 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (2m) 69s ago 34m 47.9M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 5e2e973f52fa 2024-03-21T13:06:22.953 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (11m) 69s ago 33m 76.7M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:06:22.953 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (79s) 13s ago 32m - - 19.0.0-2267-g960fbbe4 19e12293512f d404df3853a8 2024-03-21T13:06:22.953 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (73s) 69s ago 32m 74.9M - 19.0.0-2267-g960fbbe4 19e12293512f d69073373deb 2024-03-21T13:06:23.256 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:22 smithi172 ceph-mon[84635]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:06:23.257 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:22 smithi172 ceph-mon[90110]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:06:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:22 smithi177 ceph-mon[67966]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T13:06:23.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T13:06:24.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:23 smithi172 ceph-mon[84635]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:24.023 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:23 smithi172 ceph-mon[90110]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:24.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:23 smithi177 ceph-mon[67966]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:06:25.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:24 smithi172 ceph-mon[84635]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:25.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:24 smithi172 ceph-mon[90110]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:24 smithi177 ceph-mon[67966]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T13:06:25.561 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:06:25] "GET /metrics HTTP/1.1" 200 38495 "" "Prometheus/2.43.0" 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:06:26.045 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 8 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 15 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:06:26.046 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:06:26.758 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-21T13:06:27.326 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:26 smithi172 ceph-mon[84635]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-21T13:06:27.327 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:26 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/1279383782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:06:27.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:26 smithi172 ceph-mon[90110]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-21T13:06:27.327 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:26 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/1279383782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:06:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:26 smithi177 ceph-mon[67966]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-21T13:06:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:26 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/1279383782' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:06:28.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:27 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:28.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:27 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:28.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:27 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:28.567 INFO:teuthology.orchestra.run.smithi172.stdout:wait for servicemap items w/ changing names to refresh 2024-03-21T13:06:29.213 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:28 smithi172 ceph-mon[84635]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:06:29.213 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:28 smithi172 ceph-mon[90110]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:06:29.242 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'sleep 60' 2024-03-21T13:06:29.357 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:28 smithi177 ceph-mon[67966]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T13:06:31.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:30 smithi172 ceph-mon[84635]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-21T13:06:31.076 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:30 smithi172 ceph-mon[90110]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-21T13:06:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:30 smithi177 ceph-mon[67966]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-21T13:06:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:33 smithi177 ceph-mon[67966]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:33.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:33 smithi172 ceph-mon[84635]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:33.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:33 smithi172 ceph-mon[90110]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:34.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:34.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:35.313 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:35 smithi172 ceph-mon[84635]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:35.313 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:35 smithi172 ceph-mon[90110]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:35 smithi177 ceph-mon[67966]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:06:35] "GET /metrics HTTP/1.1" 200 38498 "" "Prometheus/2.43.0" 2024-03-21T13:06:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:37 smithi177 ceph-mon[67966]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:37 smithi172 ceph-mon[84635]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:37.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:37 smithi172 ceph-mon[90110]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:06:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:38 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:38.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:38 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:38 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:39 smithi177 ceph-mon[67966]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:39 smithi172 ceph-mon[84635]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:39 smithi172 ceph-mon[90110]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:41 smithi177 ceph-mon[67966]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:41.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:41 smithi172 ceph-mon[84635]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:41.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:41 smithi172 ceph-mon[90110]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:43 smithi177 ceph-mon[67966]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:43.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:43 smithi172 ceph-mon[84635]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:43.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:43 smithi172 ceph-mon[90110]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:45 smithi172 ceph-mon[84635]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:45.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:45 smithi172 ceph-mon[90110]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:45 smithi177 ceph-mon[67966]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:45.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:06:45] "GET /metrics HTTP/1.1" 200 38498 "" "Prometheus/2.43.0" 2024-03-21T13:06:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:47 smithi177 ceph-mon[67966]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:47.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:47 smithi172 ceph-mon[84635]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:47.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:47 smithi172 ceph-mon[90110]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:48 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:48.398 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:48 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:48.398 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:48 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:49 smithi177 ceph-mon[67966]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:49.359 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:49 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:49 smithi172 ceph-mon[84635]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:49 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:49 smithi172 ceph-mon[90110]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:49.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:49 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:06:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:51 smithi177 ceph-mon[67966]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:51.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:51 smithi172 ceph-mon[84635]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:51.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:51 smithi172 ceph-mon[90110]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:06:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:53 smithi177 ceph-mon[67966]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:53 smithi172 ceph-mon[84635]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:53 smithi172 ceph-mon[90110]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:55.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:55 smithi172 ceph-mon[84635]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:06:55.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:55 smithi172 ceph-mon[90110]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:06:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:55 smithi177 ceph-mon[67966]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:06:55.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:06:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:06:55] "GET /metrics HTTP/1.1" 200 38498 "" "Prometheus/2.43.0" 2024-03-21T13:06:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:57 smithi177 ceph-mon[67966]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:06:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:57 smithi172 ceph-mon[84635]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:06:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:57 smithi172 ceph-mon[90110]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:06:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:58 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:58 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:58.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:58 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:06:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:06:59 smithi177 ceph-mon[67966]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:59.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:06:59 smithi172 ceph-mon[84635]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:06:59.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:06:59 smithi172 ceph-mon[90110]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:01 smithi177 ceph-mon[67966]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:07:01.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:01 smithi172 ceph-mon[84635]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:07:01.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:01 smithi172 ceph-mon[90110]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-21T13:07:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:03 smithi177 ceph-mon[67966]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:03.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:03 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:03.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:03 smithi172 ceph-mon[84635]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:03.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:03 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:03 smithi172 ceph-mon[90110]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:03.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:03 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:05.361 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:05 smithi177 ceph-mon[67966]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:05.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:05 smithi172 ceph-mon[84635]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:05.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:07:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:07:05] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.43.0" 2024-03-21T13:07:05.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:05 smithi172 ceph-mon[90110]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:07.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:07 smithi177 ceph-mon[67966]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:07.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:07 smithi172 ceph-mon[84635]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:07.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:07 smithi172 ceph-mon[90110]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:08.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:08 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:08.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:08 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:08.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:08 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:09.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:09 smithi177 ceph-mon[67966]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:09.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:09 smithi172 ceph-mon[90110]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:09.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:09 smithi172 ceph-mon[84635]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:10.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:10 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:10.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:10 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:10.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:10 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:07:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:11 smithi177 ceph-mon[67966]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:07:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:11 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:11 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:11.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:11 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:07:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[84635]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:07:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:11.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:07:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[90110]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:07:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:07:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:07:11.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:11 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' 2024-03-21T13:07:12.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:12 smithi177 ceph-mon[67966]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:12.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:12 smithi172 ceph-mon[84635]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:12.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:12 smithi172 ceph-mon[90110]: pgmap v378: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:15.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:14 smithi172 ceph-mon[84635]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:15.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:14 smithi172 ceph-mon[90110]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:15.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:14 smithi177 ceph-mon[67966]: pgmap v379: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:15.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:07:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:07:15] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.43.0" 2024-03-21T13:07:17.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:16 smithi172 ceph-mon[84635]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:17.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:16 smithi172 ceph-mon[90110]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:17.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:16 smithi177 ceph-mon[67966]: pgmap v380: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:18.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:17 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:18.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:17 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:18.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:17 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:18 smithi177 ceph-mon[67966]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:19.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:18 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:18 smithi172 ceph-mon[84635]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:19.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:18 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:18 smithi172 ceph-mon[90110]: pgmap v381: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:19.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:18 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:21.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:20 smithi177 ceph-mon[67966]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:21.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:20 smithi172 ceph-mon[84635]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:21.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:20 smithi172 ceph-mon[90110]: pgmap v382: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:23.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:22 smithi177 ceph-mon[67966]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:23.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:22 smithi172 ceph-mon[84635]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:23.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:22 smithi172 ceph-mon[90110]: pgmap v383: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:25.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:25 smithi172 ceph-mon[84635]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:25.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:25 smithi172 ceph-mon[90110]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:25.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:25 smithi177 ceph-mon[67966]: pgmap v384: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:25.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:07:25 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:07:25] "GET /metrics HTTP/1.1" 200 38500 "" "Prometheus/2.43.0" 2024-03-21T13:07:27.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:27 smithi177 ceph-mon[67966]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:27.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:27 smithi172 ceph-mon[84635]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:27.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:27 smithi172 ceph-mon[90110]: pgmap v385: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:28.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:28 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:28.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:28 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:28.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:28 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:29.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:29 smithi177 ceph-mon[67966]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:29.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:29 smithi172 ceph-mon[84635]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:29.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:29 smithi172 ceph-mon[90110]: pgmap v386: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:31.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:31 smithi177 ceph-mon[67966]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:07:31.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:31 smithi172 ceph-mon[84635]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:07:31.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:31 smithi172 ceph-mon[90110]: pgmap v387: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T13:07:31.970 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ps' 2024-03-21T13:07:33.222 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:33 smithi172 ceph-mon[84635]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:33.222 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:33 smithi172 ceph-mon[90110]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:33.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:33 smithi177 ceph-mon[67966]: pgmap v388: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:alertmanager.a smithi172 *:9093,9094 running (12m) 84s ago 34m - - 0.25.0 c8568f914cd2 199677338a0e 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:grafana.a smithi177 *:3000 running (11m) 2m ago 33m 77.0M - 9.4.12 2bacad6d85d8 49853685e4f5 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:iscsi.foo.smithi172.bkzkmk smithi172 running (88s) 84s ago 33m - - 3.7 19e12293512f e7ee6631eaed 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.x smithi177 *:8443,9283,8765 running (13m) 2m ago 38m 454M - 19.0.0-2267-g960fbbe4 19e12293512f c7a18aa3e620 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:mgr.y smithi172 *:8443,9283,8765 running (23m) 84s ago 40m - - 19.0.0-2267-g960fbbe4 19e12293512f df80529fec67 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:mon.a smithi172 running (9m) 84s ago 40m - 2048M 19.0.0-2267-g960fbbe4 19e12293512f fb5500dccf6a 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:mon.b smithi177 running (10m) 2m ago 38m 50.9M 2048M 19.0.0-2267-g960fbbe4 19e12293512f bb78c25a6375 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:mon.c smithi172 running (9m) 84s ago 38m - 2048M 19.0.0-2267-g960fbbe4 19e12293512f 7ee992088408 2024-03-21T13:07:33.967 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.a smithi172 *:9100 running (12m) 84s ago 34m - - 1.5.0 0da6a335fe13 ffc0c5638d82 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:node-exporter.b smithi177 *:9100 running (12m) 2m ago 34m 15.3M - 1.5.0 0da6a335fe13 d7322f032b4b 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.0 smithi172 running (7m) 84s ago 38m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f bc0731501837 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.1 smithi172 running (7m) 84s ago 37m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f 76371179833d 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.2 smithi172 running (6m) 84s ago 37m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f 27e3504b1fb1 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.3 smithi172 running (5m) 84s ago 37m - 2992M 19.0.0-2267-g960fbbe4 19e12293512f 1f8f657c0c4a 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.4 smithi177 running (4m) 2m ago 36m 60.2M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 08a1127affac 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.5 smithi177 running (4m) 2m ago 36m 49.4M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 290a5061e447 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.6 smithi177 running (3m) 2m ago 35m 49.8M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 98364575cb94 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:osd.7 smithi177 running (3m) 2m ago 35m 47.9M 3248M 19.0.0-2267-g960fbbe4 19e12293512f 5e2e973f52fa 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:prometheus.a smithi177 *:9095 running (12m) 2m ago 34m 76.7M - 2.43.0 a07b618ecd1d b7238dde9e6b 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi172.aecdpn smithi172 *:8000 running (2m) 84s ago 33m - - 19.0.0-2267-g960fbbe4 19e12293512f d404df3853a8 2024-03-21T13:07:33.968 INFO:teuthology.orchestra.run.smithi172.stdout:rgw.foo.smithi177.magnlc smithi177 *:8000 running (2m) 2m ago 33m 74.9M - 19.0.0-2267-g960fbbe4 19e12293512f d69073373deb 2024-03-21T13:07:34.288 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:34 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:34.288 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:34 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:34.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:34 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:34.631 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions' 2024-03-21T13:07:35.260 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:35 smithi172 ceph-mon[84635]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:35.260 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:35 smithi172 ceph-mon[84635]: from='client.64891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:35.261 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:35 smithi172 ceph-mon[90110]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:35.261 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:35 smithi172 ceph-mon[90110]: from='client.64891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:35 smithi177 ceph-mon[67966]: pgmap v389: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:35.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:35 smithi177 ceph-mon[67966]: from='client.64891 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:35.646 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:07:35 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:07:35] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.43.0" 2024-03-21T13:07:36.779 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:07:36.779 INFO:teuthology.orchestra.run.smithi172.stdout: "mon": { 2024-03-21T13:07:36.779 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 3 2024-03-21T13:07:36.779 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:07:36.779 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr": { 2024-03-21T13:07:36.779 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: "osd": { 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 8 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw": { 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 2 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: }, 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: "overall": { 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)": 15 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout: } 2024-03-21T13:07:36.780 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:07:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:37 smithi177 ceph-mon[67966]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:37.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:37 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/3237297420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:37 smithi172 ceph-mon[84635]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:37.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:37 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/3237297420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:37 smithi172 ceph-mon[90110]: pgmap v390: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:37.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:37 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/3237297420' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:37.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch upgrade status' 2024-03-21T13:07:38.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:38 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:38.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:38 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:38.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:38 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:39.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:39 smithi177 ceph-mon[67966]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:39.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:39 smithi172 ceph-mon[84635]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:39.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:39 smithi172 ceph-mon[90110]: pgmap v391: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93", 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "in_progress": true, 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "services_complete": [ 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "rgw", 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "osd", 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "iscsi", 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "mon", 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: "mgr" 2024-03-21T13:07:39.492 INFO:teuthology.orchestra.run.smithi172.stdout: ], 2024-03-21T13:07:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: "progress": "16/21 daemons upgraded", 2024-03-21T13:07:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed.", 2024-03-21T13:07:39.493 INFO:teuthology.orchestra.run.smithi172.stdout: "is_paused": true 2024-03-21T13:07:39.493 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:07:40.161 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph health detail' 2024-03-21T13:07:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:41 smithi172 ceph-mon[90110]: from='client.54972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:41.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:41 smithi172 ceph-mon[90110]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:41 smithi172 ceph-mon[84635]: from='client.54972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:41.147 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:41 smithi172 ceph-mon[84635]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:41 smithi177 ceph-mon[67966]: from='client.54972 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:41.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:41 smithi177 ceph-mon[67966]: pgmap v392: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:42.279 INFO:teuthology.orchestra.run.smithi172.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. 2024-03-21T13:07:42.279 INFO:teuthology.orchestra.run.smithi172.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi172.bkzkmk on host smithi172 failed. 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrade daemon: iscsi.foo.smithi172.bkzkmk: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi172.bkzkmk ... 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:Creating ceph-iscsi config... 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/iscsi-gateway.cfg 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:Write file: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/iscsi.foo.smithi172.bkzkmk/tcmu-runner-entrypoint.sh 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3bbfe794\x2de77e\x2d11ee\x2d95cd\x2d87774f69a715.slice/ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl start ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:07:42.280 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout:Traceback (most recent call last): 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: exec(code, run_globals) 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5606, in 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 5594, in main 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3094, in command_deploy_from 2024-03-21T13:07:42.281 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3129, in _common_deploy 2024-03-21T13:07:42.282 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 3149, in _deploy_daemon_container 2024-03-21T13:07:42.282 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1085, in deploy_daemon 2024-03-21T13:07:42.282 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/__main__.py", line 1221, in deploy_daemon_units 2024-03-21T13:07:42.282 INFO:teuthology.orchestra.run.smithi172.stdout: File "/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/cephadm.383ed01f9644af9191b0aa3f2af2794987dfb7e7b16cccba9429bd2937190c6b/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-21T13:07:42.282 INFO:teuthology.orchestra.run.smithi172.stdout:RuntimeError: Failed command: systemctl start ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk: Job for ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service failed because the control process exited with error code. 2024-03-21T13:07:42.282 INFO:teuthology.orchestra.run.smithi172.stdout:See "systemctl status ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" and "journalctl -xeu ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@iscsi.foo.smithi172.bkzkmk.service" for details. 2024-03-21T13:07:42.968 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-21T13:07:43.276 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:43 smithi172 ceph-mon[84635]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:43.277 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:43 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/163597419' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:07:43.277 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:43 smithi172 ceph-mon[90110]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:43.277 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:43 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/163597419' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:07:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:43 smithi177 ceph-mon[67966]: pgmap v393: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:43.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:43 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/163597419' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-21T13:07:45.067 INFO:teuthology.orchestra.run.smithi172.stdout:true 2024-03-21T13:07:45.312 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:45 smithi172 ceph-mon[84635]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:45.312 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:45 smithi172 ceph-mon[90110]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:45.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:45 smithi177 ceph-mon[67966]: pgmap v394: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:45.633 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:07:45 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:07:45] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.43.0" 2024-03-21T13:07:45.659 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-21T13:07:46.216 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:46 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/680410524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:46.217 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:46 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/680410524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:46.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:46 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/680410524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:47.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:47 smithi172 ceph-mon[84635]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:47.284 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:47 smithi172 ceph-mon[90110]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:47.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:47 smithi177 ceph-mon[67966]: pgmap v395: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:47.842 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph version 19.0.0-2267-g960fbbe4 (960fbbe4645c2becc8084669be194c95183c7753) squid (dev)" 2024-03-21T13:07:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:48 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:48 smithi172 ceph-mon[84635]: from='client.? 172.21.15.172:0/2195459880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:48.146 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:48 smithi172 ceph-mon[84635]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:48.146 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:48 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:48 smithi172 ceph-mon[90110]: from='client.? 172.21.15.172:0/2195459880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:48.147 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:48 smithi172 ceph-mon[90110]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:48 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:48 smithi177 ceph-mon[67966]: from='client.? 172.21.15.172:0/2195459880' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-21T13:07:48.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:48 smithi177 ceph-mon[67966]: from='mgr.54108 172.21.15.172:0/3576381916' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T13:07:48.447 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -e sha1=960fbbe4645c2becc8084669be194c95183c7753 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-21T13:07:49.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:49 smithi177 ceph-mon[67966]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:49.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:49 smithi172 ceph-mon[84635]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:49.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:49 smithi172 ceph-mon[90110]: pgmap v396: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:50.443 INFO:teuthology.orchestra.run.smithi172.stdout:osd 8 2m ago - 2024-03-21T13:07:51.053 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-21T13:07:51.107 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi172.front.sepia.ceph.com 2024-03-21T13:07:51.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-21T13:07:51.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:51 smithi177 ceph-mon[67966]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:51.397 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:51 smithi172 ceph-mon[84635]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:51.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:51 smithi172 ceph-mon[90110]: pgmap v397: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:52.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:52 smithi177 ceph-mon[67966]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:52.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:52 smithi172 ceph-mon[84635]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:52.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:52 smithi172 ceph-mon[90110]: from='client.64927 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:53.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:53 smithi177 ceph-mon[67966]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:53.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:53 smithi172 ceph-mon[84635]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:53.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:53 smithi172 ceph-mon[90110]: pgmap v398: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:53.765 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-21T13:07:53.765 INFO:teuthology.orchestra.run.smithi172.stdout: "image": "quay.io/ceph/ceph", 2024-03-21T13:07:53.765 INFO:teuthology.orchestra.run.smithi172.stdout: "registry": "quay.io", 2024-03-21T13:07:53.765 INFO:teuthology.orchestra.run.smithi172.stdout: "bare_image": "ceph/ceph", 2024-03-21T13:07:53.766 INFO:teuthology.orchestra.run.smithi172.stdout: "versions": [] 2024-03-21T13:07:53.766 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-21T13:07:54.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:54 smithi177 ceph-mon[67966]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:54.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:54 smithi172 ceph-mon[84635]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:54.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:54 smithi172 ceph-mon[90110]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:54.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-21T13:07:55.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:55 smithi177 ceph-mon[67966]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:55.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:55 smithi172 ceph-mon[84635]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:55.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:55 smithi172 ceph-mon[90110]: pgmap v399: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:55.396 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:07:55 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: ::ffff:172.21.15.177 - - [21/Mar/2024:13:07:55] "GET /metrics HTTP/1.1" 200 38500 "" "Prometheus/2.43.0" 2024-03-21T13:07:57.130 INFO:teuthology.orchestra.run.smithi172.stdout: "16.2.0", 2024-03-21T13:07:57.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:57 smithi177 ceph-mon[67966]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:57.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:57 smithi172 ceph-mon[90110]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:57.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:57 smithi172 ceph-mon[84635]: pgmap v400: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:07:57.753 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-21T13:07:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:58 smithi177 ceph-mon[67966]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:58.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:58 smithi177 ceph-mon[67966]: from='client.55008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:58 smithi172 ceph-mon[84635]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:58.396 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:58 smithi172 ceph-mon[84635]: from='client.55008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:58 smithi172 ceph-mon[90110]: from='client.44642 -' entity='client.iscsi.foo.smithi172.bkzkmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T13:07:58.397 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:58 smithi172 ceph-mon[90110]: from='client.55008 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:07:59.292 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:07:59 smithi172 ceph-mon[84635]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:59.292 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:07:59 smithi172 ceph-mon[90110]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:07:59.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:07:59 smithi177 ceph-mon[67966]: pgmap v401: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T13:08:00.326 INFO:teuthology.orchestra.run.smithi172.stdout: "v16.2.2", 2024-03-21T13:08:00.327 INFO:teuthology.orchestra.run.smithi172.stdout: "v16.2.2-20210505", 2024-03-21T13:08:00.949 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-21T13:08:00.969 INFO:tasks.cephadm:Teardown begin 2024-03-21T13:08:00.969 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:08:01.001 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:08:01.037 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-21T13:08:01.037 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:08:01.057 DEBUG:teuthology.orchestra.run.smithi177:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:08:01.094 INFO:tasks.cephadm:Stopping all daemons... 2024-03-21T13:08:01.095 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-21T13:08:01.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a 2024-03-21T13:08:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:01 smithi177 ceph-mon[67966]: from='client.64945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:01.358 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:01 smithi177 ceph-mon[67966]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:08:01.386 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-mon[84635]: from='client.64945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:01.386 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-mon[84635]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:08:01.386 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 systemd[1]: Stopping Ceph mon.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:01.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-mon[90110]: from='client.64945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:08:01.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-mon[90110]: pgmap v402: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T13:08:01.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a[84631]: 2024-03-21T13:08:01.385+0000 7f776c066700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:01.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a[84631]: 2024-03-21T13:08:01.385+0000 7f776c066700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-21T13:08:01.646 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 podman[137229]: 2024-03-21 13:08:01.477225893 +0000 UTC m=+0.190613751 container died fb5500dccf6a39ba5783434c0cab7f82ab8c1105da245175efc6fb8e6cb7188d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:13:08:01] ENGINE Bus STOPPING 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:13:08:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:13:08:01] ENGINE Bus STOPPED 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:13:08:01] ENGINE Bus STARTING 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: CherryPy Checker: 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: The Application mounted at '' has an empty config. 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:13:08:01] ENGINE Serving on http://:::9283 2024-03-21T13:08:01.647 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:01 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: [21/Mar/2024:13:08:01] ENGINE Bus STARTED 2024-03-21T13:08:02.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 podman[137229]: 2024-03-21 13:08:01.836081387 +0000 UTC m=+0.549469247 container cleanup fb5500dccf6a39ba5783434c0cab7f82ab8c1105da245175efc6fb8e6cb7188d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-21T13:08:02.112 INFO:journalctl@ceph.mon.a.smithi172.stdout:Mar 21 13:08:01 smithi172 bash[137229]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-a 2024-03-21T13:08:02.261 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.a.service' 2024-03-21T13:08:02.314 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:02.314 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-21T13:08:02.314 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-21T13:08:02.314 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.c 2024-03-21T13:08:02.396 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:02 smithi172 systemd[1]: Stopping Ceph mon.c for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c[90106]: 2024-03-21T13:08:02.639+0000 7f4e70a83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:02.896 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:02 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c[90106]: 2024-03-21T13:08:02.640+0000 7f4e70a83700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-21T13:08:02.897 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:02 smithi172 podman[137363]: 2024-03-21 13:08:02.762848792 +0000 UTC m=+0.233684333 container died 7ee99208840831e894a23a5e8279e33d0df284379739b1384eb73028b98dd1ce (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T13:08:03.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:03 smithi172 podman[137363]: 2024-03-21 13:08:03.123657228 +0000 UTC m=+0.594492756 container cleanup 7ee99208840831e894a23a5e8279e33d0df284379739b1384eb73028b98dd1ce (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:08:03.366 INFO:journalctl@ceph.mon.c.smithi172.stdout:Mar 21 13:08:03 smithi172 bash[137363]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-c 2024-03-21T13:08:03.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.c.service' 2024-03-21T13:08:03.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:03.529 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-21T13:08:03.529 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-21T13:08:03.529 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.b 2024-03-21T13:08:03.858 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:03 smithi177 systemd[1]: Stopping Ceph mon.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:04.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b[67962]: 2024-03-21T13:08:03.913+0000 7f25d4483700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:04.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b[67962]: 2024-03-21T13:08:03.913+0000 7f25d4483700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-21T13:08:04.110 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:03 smithi177 podman[101056]: 2024-03-21 13:08:03.934273756 +0000 UTC m=+0.202319065 container died bb78c25a637538feab6e0ee583e5c91fcac41b2f366c3621e1f8ed6f6c4a1413 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-21T13:08:04.706 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:04 smithi177 podman[101056]: 2024-03-21 13:08:04.396806556 +0000 UTC m=+0.664851862 container cleanup bb78c25a637538feab6e0ee583e5c91fcac41b2f366c3621e1f8ed6f6c4a1413 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T13:08:04.706 INFO:journalctl@ceph.mon.b.smithi177.stdout:Mar 21 13:08:04 smithi177 bash[101056]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mon-b 2024-03-21T13:08:04.898 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mon.b.service' 2024-03-21T13:08:04.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:04.951 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-21T13:08:04.951 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-21T13:08:04.951 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y 2024-03-21T13:08:05.253 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:04 smithi172 systemd[1]: Stopping Ceph mgr.y for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:05.554 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:05 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y[63153]: 2024-03-21T13:08:05.253+0000 7f0d45476700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:05.554 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:05 smithi172 podman[137496]: 2024-03-21 13:08:05.294254155 +0000 UTC m=+0.158301244 container died df80529fec67903ea1c3a261037832a943d5f31c81cf90231023380b6ddc6d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T13:08:05.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:05 smithi172 podman[137496]: 2024-03-21 13:08:05.645110845 +0000 UTC m=+0.509157926 container cleanup df80529fec67903ea1c3a261037832a943d5f31c81cf90231023380b6ddc6d8e (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:08:05.896 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Mar 21 13:08:05 smithi172 bash[137496]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-y 2024-03-21T13:08:06.058 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-21T13:08:06.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:06.097 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-21T13:08:06.097 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-21T13:08:06.097 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x 2024-03-21T13:08:06.440 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 13:08:06 smithi177 systemd[1]: Stopping Ceph mgr.x for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:06.727 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 13:08:06 smithi177 podman[101192]: 2024-03-21 13:08:06.440364115 +0000 UTC m=+0.139159211 container died c7a18aa3e6205f001da96c22b6e045db187aa359d298ef6929db7fbfd2b7c8fd (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:08:07.069 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 13:08:06 smithi177 podman[101192]: 2024-03-21 13:08:06.809987159 +0000 UTC m=+0.508782257 container cleanup c7a18aa3e6205f001da96c22b6e045db187aa359d298ef6929db7fbfd2b7c8fd (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-21T13:08:07.069 INFO:journalctl@ceph.mgr.x.smithi177.stdout:Mar 21 13:08:06 smithi177 bash[101192]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-mgr-x 2024-03-21T13:08:07.229 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-21T13:08:07.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:07.266 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-21T13:08:07.267 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-21T13:08:07.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.0 2024-03-21T13:08:07.582 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:07 smithi172 systemd[1]: Stopping Ceph osd.0 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:07.896 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:07 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T13:08:07.581+0000 7f535d533700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:07.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:07 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T13:08:07.581+0000 7f535d533700 -1 osd.0 159 *** Got signal Terminated *** 2024-03-21T13:08:07.897 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:07 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0[98366]: 2024-03-21T13:08:07.581+0000 7f535d533700 -1 osd.0 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:13.225 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:12 smithi172 podman[137617]: 2024-03-21 13:08:12.944575665 +0000 UTC m=+5.481305861 container died bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:08:13.545 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:13 smithi172 podman[137617]: 2024-03-21 13:08:13.324215085 +0000 UTC m=+5.860945272 container cleanup bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:08:13.545 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:13 smithi172 bash[137617]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0 2024-03-21T13:08:13.896 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:13 smithi172 podman[137634]: 2024-03-21 13:08:13.584060528 +0000 UTC m=+0.636748233 container remove bc0731501837bb38277f198608900c6d7c9b8ed4912edbba40131fc4c8c65520 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T13:08:14.203 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:13 smithi172 podman[137726]: 2024-03-21 13:08:13.918042404 +0000 UTC m=+0.090610407 container create b58b514c7e6ad50738380fb13bd3c7e801433af5c365458b6cfdea034ed3db71 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-21T13:08:14.204 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:13 smithi172 podman[137726]: 2024-03-21 13:08:13.844962989 +0000 UTC m=+0.017531000 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:08:14.204 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:14 smithi172 podman[137726]: 2024-03-21 13:08:14.128929558 +0000 UTC m=+0.301497562 container init b58b514c7e6ad50738380fb13bd3c7e801433af5c365458b6cfdea034ed3db71 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T13:08:14.204 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:14 smithi172 podman[137726]: 2024-03-21 13:08:14.132234948 +0000 UTC m=+0.304802956 container start b58b514c7e6ad50738380fb13bd3c7e801433af5c365458b6cfdea034ed3db71 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-21T13:08:14.204 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:14 smithi172 podman[137726]: 2024-03-21 13:08:14.203835592 +0000 UTC m=+0.376403604 container attach b58b514c7e6ad50738380fb13bd3c7e801433af5c365458b6cfdea034ed3db71 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T13:08:14.646 INFO:journalctl@ceph.osd.0.smithi172.stdout:Mar 21 13:08:14 smithi172 podman[137726]: 2024-03-21 13:08:14.432998768 +0000 UTC m=+0.605566762 container died b58b514c7e6ad50738380fb13bd3c7e801433af5c365458b6cfdea034ed3db71 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:08:15.082 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.0.service' 2024-03-21T13:08:15.125 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:15.125 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-21T13:08:15.125 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-21T13:08:15.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.1 2024-03-21T13:08:15.320 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:15 smithi172 systemd[1]: Stopping Ceph osd.1 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:15.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:08:15.478+0000 7f6aeb19a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:15.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:08:15.478+0000 7f6aeb19a700 -1 osd.1 159 *** Got signal Terminated *** 2024-03-21T13:08:15.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1[103259]: 2024-03-21T13:08:15.478+0000 7f6aeb19a700 -1 osd.1 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:21.245 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:20 smithi172 podman[137831]: 2024-03-21 13:08:20.96948954 +0000 UTC m=+5.613000091 container died 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-21T13:08:21.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:21 smithi172 podman[137831]: 2024-03-21 13:08:21.452097297 +0000 UTC m=+6.095607857 container cleanup 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-03-21T13:08:21.646 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:21 smithi172 bash[137831]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1 2024-03-21T13:08:22.003 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:21 smithi172 podman[137861]: 2024-03-21 13:08:21.70365816 +0000 UTC m=+0.729674957 container remove 76371179833d31cc1a2a7552281c16d113aba79404b5076c5c330920a988070d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-21T13:08:22.257 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:22 smithi172 podman[137953]: 2024-03-21 13:08:22.004212802 +0000 UTC m=+0.112214368 container create 100b0b694a7fbe8b8339d30ecaf2a2b89efaa0ca442dea5753e16c939d013819 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux ) 2024-03-21T13:08:22.257 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:22 smithi172 podman[137953]: 2024-03-21 13:08:21.910275787 +0000 UTC m=+0.018277359 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:08:22.257 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:22 smithi172 podman[137953]: 2024-03-21 13:08:22.257439932 +0000 UTC m=+0.365441509 container init 100b0b694a7fbe8b8339d30ecaf2a2b89efaa0ca442dea5753e16c939d013819 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-21T13:08:22.555 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:22 smithi172 podman[137953]: 2024-03-21 13:08:22.260995146 +0000 UTC m=+0.368996720 container start 100b0b694a7fbe8b8339d30ecaf2a2b89efaa0ca442dea5753e16c939d013819 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-21T13:08:22.555 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:22 smithi172 podman[137953]: 2024-03-21 13:08:22.365318388 +0000 UTC m=+0.473319963 container attach 100b0b694a7fbe8b8339d30ecaf2a2b89efaa0ca442dea5753e16c939d013819 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-21T13:08:22.851 INFO:journalctl@ceph.osd.1.smithi172.stdout:Mar 21 13:08:22 smithi172 podman[137953]: 2024-03-21 13:08:22.555294411 +0000 UTC m=+0.663295969 container died 100b0b694a7fbe8b8339d30ecaf2a2b89efaa0ca442dea5753e16c939d013819 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T13:08:23.133 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.1.service' 2024-03-21T13:08:23.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:23.176 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-21T13:08:23.177 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T13:08:23.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.2 2024-03-21T13:08:23.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:23 smithi172 systemd[1]: Stopping Ceph osd.2 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:23.896 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:08:23.504+0000 7f247b237700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:23.896 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:08:23.504+0000 7f247b237700 -1 osd.2 159 *** Got signal Terminated *** 2024-03-21T13:08:23.896 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:23 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2[110591]: 2024-03-21T13:08:23.504+0000 7f247b237700 -1 osd.2 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:29.237 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:28 smithi172 podman[138058]: 2024-03-21 13:08:28.897137492 +0000 UTC m=+5.498492501 container died 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:08:29.646 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:29 smithi172 podman[138058]: 2024-03-21 13:08:29.237421519 +0000 UTC m=+5.838776529 container cleanup 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T13:08:29.646 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:29 smithi172 bash[138058]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2 2024-03-21T13:08:29.646 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:29 smithi172 podman[138076]: 2024-03-21 13:08:29.397230968 +0000 UTC m=+0.496193380 container remove 27e3504b1fb12a279f2fe616a234ccb9bb0f2dc278e48a02971fcd6ec05cee54 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T13:08:30.125 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:29 smithi172 podman[138171]: 2024-03-21 13:08:29.704080101 +0000 UTC m=+0.018259238 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:08:30.125 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:29 smithi172 podman[138171]: 2024-03-21 13:08:29.823159311 +0000 UTC m=+0.137338479 container create e202aa77f3758b5986a8653a932e5aa38bce4d27ad20b848ee59ae1675e45a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-21T13:08:30.125 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:30 smithi172 podman[138171]: 2024-03-21 13:08:30.042655663 +0000 UTC m=+0.356834794 container init e202aa77f3758b5986a8653a932e5aa38bce4d27ad20b848ee59ae1675e45a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True) 2024-03-21T13:08:30.125 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:30 smithi172 podman[138171]: 2024-03-21 13:08:30.046127252 +0000 UTC m=+0.360306375 container start e202aa77f3758b5986a8653a932e5aa38bce4d27ad20b848ee59ae1675e45a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T13:08:30.125 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:30 smithi172 podman[138171]: 2024-03-21 13:08:30.125372858 +0000 UTC m=+0.439551987 container attach e202aa77f3758b5986a8653a932e5aa38bce4d27ad20b848ee59ae1675e45a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:08:30.396 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:30 smithi172 podman[138171]: 2024-03-21 13:08:30.34790778 +0000 UTC m=+0.662086909 container died e202aa77f3758b5986a8653a932e5aa38bce4d27ad20b848ee59ae1675e45a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:08:30.928 INFO:journalctl@ceph.osd.2.smithi172.stdout:Mar 21 13:08:30 smithi172 podman[138196]: 2024-03-21 13:08:30.911906995 +0000 UTC m=+0.558955724 container remove e202aa77f3758b5986a8653a932e5aa38bce4d27ad20b848ee59ae1675e45a4a (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T13:08:30.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T13:08:30.973 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:30.973 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T13:08:30.974 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-21T13:08:30.974 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.3 2024-03-21T13:08:31.298 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:31 smithi172 systemd[1]: Stopping Ceph osd.3 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:31.646 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:08:31.297+0000 7f196e7dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:31.646 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:08:31.297+0000 7f196e7dc700 -1 osd.3 159 *** Got signal Terminated *** 2024-03-21T13:08:31.646 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:31 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:08:31.297+0000 7f196e7dc700 -1 osd.3 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:34.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:34 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:34.078+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:34.775 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:34 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:34.503+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:35.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:35.092+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:35.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:35 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:35.538+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:36.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:36.079+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:36.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:36.297+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:36.396 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:36 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3[117868]: 2024-03-21T13:08:36.123+0000 7f196add0700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.893741+0000 front 2024-03-21T13:08:09.893909+0000 (oldest deadline 2024-03-21T13:08:35.193919+0000) 2024-03-21T13:08:36.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:36.389+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:36.859 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:36 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:36.513+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:36.896 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:36 smithi172 podman[138275]: 2024-03-21 13:08:36.645444073 +0000 UTC m=+5.453705314 container died 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-21T13:08:37.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:37 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:37.056+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:37.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:37 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:37.282+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:37.396 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:36 smithi172 podman[138275]: 2024-03-21 13:08:36.981192704 +0000 UTC m=+5.789453984 container cleanup 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-21T13:08:37.396 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:36 smithi172 bash[138275]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3 2024-03-21T13:08:37.396 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:37 smithi172 podman[138304]: 2024-03-21 13:08:37.14060115 +0000 UTC m=+0.491912224 container remove 1f8f657c0c4a17c9d3120a68b24ea0de8df5bbbdfad499d917d5c7aed12f8764 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T13:08:37.801 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:37 smithi172 podman[138395]: 2024-03-21 13:08:37.448573322 +0000 UTC m=+0.018254217 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:08:37.802 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:37 smithi172 podman[138395]: 2024-03-21 13:08:37.557907771 +0000 UTC m=+0.127588712 container create b562d6ae0f6ab7e08f1774bc86a2c015aa529f6492ca0e8e1a31ed4b80bb5582 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:08:37.802 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:37 smithi172 podman[138395]: 2024-03-21 13:08:37.80187506 +0000 UTC m=+0.371555965 container init b562d6ae0f6ab7e08f1774bc86a2c015aa529f6492ca0e8e1a31ed4b80bb5582 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-21T13:08:37.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:37 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:37.384+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:37.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:37 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:37.524+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:38.093 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:37 smithi172 podman[138395]: 2024-03-21 13:08:37.805003851 +0000 UTC m=+0.374684755 container start b562d6ae0f6ab7e08f1774bc86a2c015aa529f6492ca0e8e1a31ed4b80bb5582 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-21T13:08:38.094 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:37 smithi172 podman[138395]: 2024-03-21 13:08:37.893510082 +0000 UTC m=+0.463190981 container attach b562d6ae0f6ab7e08f1774bc86a2c015aa529f6492ca0e8e1a31ed4b80bb5582 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T13:08:38.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:38.106+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:38.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:38.285+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:38.396 INFO:journalctl@ceph.osd.3.smithi172.stdout:Mar 21 13:08:38 smithi172 podman[138395]: 2024-03-21 13:08:38.094050236 +0000 UTC m=+0.663731140 container died b562d6ae0f6ab7e08f1774bc86a2c015aa529f6492ca0e8e1a31ed4b80bb5582 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2) 2024-03-21T13:08:38.690 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.3.service' 2024-03-21T13:08:38.734 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:38.734 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-21T13:08:38.734 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-21T13:08:38.734 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.4 2024-03-21T13:08:38.765 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:38.373+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:38.766 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:38 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:38.497+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:39.068 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:38 smithi177 systemd[1]: Stopping Ceph osd.4 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:39.328 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:39.067+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:39.328 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:39.250+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:39.608 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:39.410+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:39.608 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:39.327+0000 7fb1cd9db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:39.609 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:39.327+0000 7fb1cd9db700 -1 osd.4 159 *** Got signal Terminated *** 2024-03-21T13:08:39.609 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:39.327+0000 7fb1cd9db700 -1 osd.4 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:39.609 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:39 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:39.502+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:40.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:40.085+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:40.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:40.204+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:40.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:40.443+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:40.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:40 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:40.536+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:41.471 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:41.125+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:41.471 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:41.227+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:41.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:41.471+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:41.859 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:41.576+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:41.859 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:41.576+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:19.221707+0000 front 2024-03-21T13:08:19.221624+0000 (oldest deadline 2024-03-21T13:08:41.422001+0000) 2024-03-21T13:08:42.512 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:42.130+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:42.512 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:42.246+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:42.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:42.512+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:42.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:42.580+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:42.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:42 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:42.580+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:19.221707+0000 front 2024-03-21T13:08:19.221624+0000 (oldest deadline 2024-03-21T13:08:41.422001+0000) 2024-03-21T13:08:43.463 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:43.216+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:43.463 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:43.134+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:43.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:43.463+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:43.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:43.532+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.119547+0000 front 2024-03-21T13:08:10.119587+0000 (oldest deadline 2024-03-21T13:08:33.619371+0000) 2024-03-21T13:08:43.858 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:43 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4[78292]: 2024-03-21T13:08:43.532+0000 7fb1c5fc7700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:19.221707+0000 front 2024-03-21T13:08:19.221624+0000 (oldest deadline 2024-03-21T13:08:41.422001+0000) 2024-03-21T13:08:43.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:08:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:08:43.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T13:08:43.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:08:43 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:08:43.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T13:08:44.470 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:44.155+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:44.470 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:44.188+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:44.766 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:44 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:44.468+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:45.053 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:44 smithi177 podman[101328]: 2024-03-21 13:08:44.765938081 +0000 UTC m=+5.828808510 container died 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-21T13:08:45.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:45.180+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:45.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:45.227+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:45.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:45 smithi177 podman[101328]: 2024-03-21 13:08:45.252767386 +0000 UTC m=+6.315637852 container cleanup 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T13:08:45.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:45 smithi177 bash[101328]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4 2024-03-21T13:08:45.833 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:45 smithi177 podman[101345]: 2024-03-21 13:08:45.495907756 +0000 UTC m=+0.731180735 container remove 08a1127affac1ebdd0d59f71c2608959839cf5e8d5ed80e96e07b4eb5c3195ac (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-21T13:08:45.833 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:45.437+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:45.834 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:45 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:45.437+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:46.099 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:45 smithi177 podman[101438]: 2024-03-21 13:08:45.736658693 +0000 UTC m=+0.018332920 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:08:46.099 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:45 smithi177 podman[101438]: 2024-03-21 13:08:45.846902788 +0000 UTC m=+0.128577057 container create 209885046bcadb92cd3a0b6aab5180eb4ce64e324c8936b4cb107cea771a91f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T13:08:46.099 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:46 smithi177 podman[101438]: 2024-03-21 13:08:46.099371818 +0000 UTC m=+0.381046045 container init 209885046bcadb92cd3a0b6aab5180eb4ce64e324c8936b4cb107cea771a91f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-21T13:08:46.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:46.205+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:46.358 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:46.205+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:46.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:46 smithi177 podman[101438]: 2024-03-21 13:08:46.102883378 +0000 UTC m=+0.384557609 container start 209885046bcadb92cd3a0b6aab5180eb4ce64e324c8936b4cb107cea771a91f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T13:08:46.358 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:46 smithi177 podman[101438]: 2024-03-21 13:08:46.191101665 +0000 UTC m=+0.472775904 container attach 209885046bcadb92cd3a0b6aab5180eb4ce64e324c8936b4cb107cea771a91f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:08:46.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:46.276+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:46.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:46.276+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:46.678 INFO:journalctl@ceph.osd.4.smithi177.stdout:Mar 21 13:08:46 smithi177 podman[101438]: 2024-03-21 13:08:46.400696444 +0000 UTC m=+0.682370672 container died 209885046bcadb92cd3a0b6aab5180eb4ce64e324c8936b4cb107cea771a91f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-21T13:08:46.679 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:46.419+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:46.679 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:46 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:46.419+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:46.955 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.4.service' 2024-03-21T13:08:47.037 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:47.037 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-21T13:08:47.037 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-21T13:08:47.037 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.5 2024-03-21T13:08:47.200 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:47 smithi177 systemd[1]: Stopping Ceph osd.5 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:47.201 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:47.198+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:47.201 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:47.198+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:47.578 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:47.381+0000 7f4276c86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:47.578 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:47.381+0000 7f4276c86700 -1 osd.5 159 *** Got signal Terminated *** 2024-03-21T13:08:47.578 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:47.381+0000 7f4276c86700 -1 osd.5 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:47.579 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:47.375+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:47.579 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:47.375+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:47.579 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:47.264+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:47.579 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:47 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:47.264+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:48.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:48.184+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:48.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:48.184+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:48.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:48.281+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:48.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:48.281+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:48.609 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:48.354+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:48.609 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:48 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:48.354+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:49.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:49 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:49.151+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:49.608 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:49 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:49.151+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:49.608 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:49 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:49.392+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:49.608 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:49 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:49.392+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:49.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:49 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:49.278+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:49.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:49 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:49.278+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:50.413 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:50.275+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:50.414 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:50.275+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:50.414 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:50.119+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:50.414 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:50.119+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:50.414 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:50.119+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:27.713793+0000 front 2024-03-21T13:08:27.713761+0000 (oldest deadline 2024-03-21T13:08:49.413509+0000) 2024-03-21T13:08:50.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:50.413+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:50.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:50 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:50.413+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:51.464 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:51.286+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:51.464 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:51.286+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:51.464 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:51.155+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:51.464 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:51.155+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:51.464 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:51.155+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:27.713793+0000 front 2024-03-21T13:08:27.713761+0000 (oldest deadline 2024-03-21T13:08:49.413509+0000) 2024-03-21T13:08:51.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:51.463+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:51.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:51.463+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:51.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:51 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:51.463+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:52.493 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:52.185+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:11.010819+0000 front 2024-03-21T13:08:11.010805+0000 (oldest deadline 2024-03-21T13:08:33.310592+0000) 2024-03-21T13:08:52.493 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:52.185+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.112799+0000 front 2024-03-21T13:08:20.112765+0000 (oldest deadline 2024-03-21T13:08:45.412654+0000) 2024-03-21T13:08:52.493 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5[83129]: 2024-03-21T13:08:52.185+0000 7f427327a700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:27.713793+0000 front 2024-03-21T13:08:27.713761+0000 (oldest deadline 2024-03-21T13:08:49.413509+0000) 2024-03-21T13:08:52.494 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:52.308+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:52.494 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:52.308+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:52.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:52.492+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:52.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:52.492+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:52.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:52 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:52.492+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:53.191 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:52 smithi177 podman[101555]: 2024-03-21 13:08:52.894099054 +0000 UTC m=+5.631539144 container died 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:08:53.503 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:53.320+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:53.503 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:53.320+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:53.504 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:53 smithi177 podman[101555]: 2024-03-21 13:08:53.364831537 +0000 UTC m=+6.102271634 container cleanup 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T13:08:53.504 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:53 smithi177 bash[101555]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5 2024-03-21T13:08:53.858 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:53 smithi177 podman[101576]: 2024-03-21 13:08:53.649809578 +0000 UTC m=+0.745803233 container remove 290a5061e447def0ceadc72bf8eeb676ab1a212c463a8d06ae6c79a124eaa47d (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-21T13:08:53.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:53.502+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:53.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:53.502+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:53.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:53 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:53.502+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:08:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:08:53.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T13:08:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:08:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:08:53.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T13:08:53.896 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:08:53 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:08:53.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T13:08:54.272 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:53 smithi177 podman[101668]: 2024-03-21 13:08:53.893421385 +0000 UTC m=+0.019001278 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:08:54.272 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:54 smithi177 podman[101668]: 2024-03-21 13:08:54.000601829 +0000 UTC m=+0.126181693 container create d7212a8b71e48e67d50948d51818961cec09065789d1e7a2f4009b5e335e423f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:08:54.272 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:54 smithi177 podman[101668]: 2024-03-21 13:08:54.211769408 +0000 UTC m=+0.337349280 container init d7212a8b71e48e67d50948d51818961cec09065789d1e7a2f4009b5e335e423f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:08:54.272 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:54 smithi177 podman[101668]: 2024-03-21 13:08:54.214796276 +0000 UTC m=+0.340376146 container start d7212a8b71e48e67d50948d51818961cec09065789d1e7a2f4009b5e335e423f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-21T13:08:54.527 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:54 smithi177 podman[101668]: 2024-03-21 13:08:54.303271889 +0000 UTC m=+0.428851780 container attach d7212a8b71e48e67d50948d51818961cec09065789d1e7a2f4009b5e335e423f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:08:54.527 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:54 smithi177 podman[101668]: 2024-03-21 13:08:54.508106966 +0000 UTC m=+0.633686827 container died d7212a8b71e48e67d50948d51818961cec09065789d1e7a2f4009b5e335e423f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-21T13:08:54.527 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:54.271+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:54.527 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:54.271+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:54.528 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:54.271+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:08:54.798 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:54.525+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:54.799 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:54.525+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:54.799 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:54 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:54.525+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:55.055 INFO:journalctl@ceph.osd.5.smithi177.stdout:Mar 21 13:08:55 smithi177 podman[101694]: 2024-03-21 13:08:55.040402599 +0000 UTC m=+0.526911189 container remove d7212a8b71e48e67d50948d51818961cec09065789d1e7a2f4009b5e335e423f (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-21T13:08:55.061 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.5.service' 2024-03-21T13:08:55.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:08:55.104 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-21T13:08:55.105 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T13:08:55.105 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.6 2024-03-21T13:08:55.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 systemd[1]: Stopping Ceph osd.6 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:08:55.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:55.254+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:55.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:55.254+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:55.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:55.254+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:08:55.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:55.459+0000 7f7ce64c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:08:55.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:55.459+0000 7f7ce64c8700 -1 osd.6 159 *** Got signal Terminated *** 2024-03-21T13:08:55.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:55.459+0000 7f7ce64c8700 -1 osd.6 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:08:55.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:55.542+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:55.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:55.542+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:55.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:55 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:55.542+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:56.545 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:56.234+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:56.545 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:56.234+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:56.545 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:56.234+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:08:56.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:56.544+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:56.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:56.544+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:56.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:56 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:56.544+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:57.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:57 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:57.199+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:57.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:57 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:57.199+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:57.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:57 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:57.199+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:08:57.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:57 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:57.582+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:57.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:57 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:57.582+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:57.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:57 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:57.582+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:58.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:58.237+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:58.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:58.237+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:58.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:58.237+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:08:58.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:58.237+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:08:59.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:58.630+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:59.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:58.630+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:59.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:58 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:58.630+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:59.598 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:59.207+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:08:59.598 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:59.207+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:08:59.599 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:59.207+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:08:59.599 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:59.207+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:08:59.599 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:08:59.207+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:08:59.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:59.592+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:09.765932+0000 front 2024-03-21T13:08:09.765964+0000 (oldest deadline 2024-03-21T13:08:35.665696+0000) 2024-03-21T13:08:59.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:59.592+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.366766+0000 front 2024-03-21T13:08:20.366694+0000 (oldest deadline 2024-03-21T13:08:45.066672+0000) 2024-03-21T13:08:59.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:59.592+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:26.667996+0000 front 2024-03-21T13:08:26.668025+0000 (oldest deadline 2024-03-21T13:08:51.367886+0000) 2024-03-21T13:08:59.858 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:08:59 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6[87855]: 2024-03-21T13:08:59.592+0000 7f7ce2abc700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:34.268826+0000 front 2024-03-21T13:08:34.268790+0000 (oldest deadline 2024-03-21T13:08:59.568786+0000) 2024-03-21T13:09:00.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:00.212+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:00.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:00.212+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:00.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:00.212+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:00.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:00.212+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:00.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:00 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:00.213+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:01.096 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:00 smithi177 podman[101773]: 2024-03-21 13:09:00.837334024 +0000 UTC m=+5.510133153 container died 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe) 2024-03-21T13:09:01.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:01 smithi177 podman[101773]: 2024-03-21 13:09:01.19478254 +0000 UTC m=+5.867581669 container cleanup 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T13:09:01.358 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:01 smithi177 bash[101773]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6 2024-03-21T13:09:01.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:01.241+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:01.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:01.241+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:01.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:01.241+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:01.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:01.241+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:01.359 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:01 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:01.241+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:01.798 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:01 smithi177 podman[101791]: 2024-03-21 13:09:01.45432577 +0000 UTC m=+0.619698319 container remove 98364575cb94e3368c6a82f762c66249926049a07082a0f99b4d71149106da30 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-21T13:09:02.108 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:01 smithi177 podman[101884]: 2024-03-21 13:09:01.701845025 +0000 UTC m=+0.017751322 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:09:02.109 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:01 smithi177 podman[101884]: 2024-03-21 13:09:01.821929438 +0000 UTC m=+0.137835713 container create db9f9bb1e611cd62f09262435822e41f8707377395525f833a30adbc3e47ed56 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T13:09:02.109 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:02 smithi177 podman[101884]: 2024-03-21 13:09:02.024505393 +0000 UTC m=+0.340411675 container init db9f9bb1e611cd62f09262435822e41f8707377395525f833a30adbc3e47ed56 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:09:02.109 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:02 smithi177 podman[101884]: 2024-03-21 13:09:02.027731858 +0000 UTC m=+0.343638132 container start db9f9bb1e611cd62f09262435822e41f8707377395525f833a30adbc3e47ed56 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:02 smithi177 podman[101884]: 2024-03-21 13:09:02.129104572 +0000 UTC m=+0.445010854 container attach db9f9bb1e611cd62f09262435822e41f8707377395525f833a30adbc3e47ed56 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.6.smithi177.stdout:Mar 21 13:09:02 smithi177 podman[101884]: 2024-03-21 13:09:02.319910329 +0000 UTC m=+0.635816592 container died db9f9bb1e611cd62f09262435822e41f8707377395525f833a30adbc3e47ed56 (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_BRANCH=HEAD) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:02.253+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:02.253+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:02.253+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:02.253+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:02.560 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:02 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:02.253+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:02.867 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T13:09:02.907 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:09:02.907 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T13:09:02.907 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-21T13:09:02.907 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.7 2024-03-21T13:09:03.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:02 smithi177 systemd[1]: Stopping Ceph osd.7 for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:09:03.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.222+0000 7f5203529700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:09:03.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.222+0000 7f5203529700 -1 osd.7 159 *** Got signal Terminated *** 2024-03-21T13:09:03.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.222+0000 7f5203529700 -1 osd.7 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T13:09:03.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.272+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:03.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.272+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:03.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.272+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:03.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.272+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:03.609 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:03 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:03.272+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:03.807 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:03.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T13:09:03.807 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:03.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T13:09:03.807 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:03 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:03.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T13:09:04.510 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:04.236+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:04.511 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:04.236+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:04.511 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:04.236+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:04.511 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:04.236+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:04.511 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:04.236+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:04.817Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-21T13:09:05.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:04.820Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-21T13:09:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:04.820Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-21T13:09:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:04.820Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-21T13:09:05.109 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:04 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:04.820Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-21T13:09:05.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:05.201+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:05.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:05.201+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:05.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:05.201+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:05.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:05.201+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:05.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:05 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:05.202+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:06.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:06.213+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:06.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:06.213+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:06.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:06.213+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:06.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:06.213+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:06.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:06 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:06.213+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:07.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:07.248+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2024-03-21T13:08:10.473316+0000 front 2024-03-21T13:08:10.473200+0000 (oldest deadline 2024-03-21T13:08:35.773025+0000) 2024-03-21T13:09:07.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:07.248+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2024-03-21T13:08:20.473839+0000 front 2024-03-21T13:08:20.473899+0000 (oldest deadline 2024-03-21T13:08:45.773752+0000) 2024-03-21T13:09:07.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:07.248+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2024-03-21T13:08:28.074786+0000 front 2024-03-21T13:08:28.074890+0000 (oldest deadline 2024-03-21T13:08:53.374611+0000) 2024-03-21T13:09:07.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:07.248+0000 7f51ffb1d700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2024-03-21T13:08:33.374988+0000 front 2024-03-21T13:08:33.375033+0000 (oldest deadline 2024-03-21T13:08:57.475060+0000) 2024-03-21T13:09:07.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:07 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7[92409]: 2024-03-21T13:09:07.248+0000 7f51ffb1d700 -1 osd.7 159 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.54702.0:42 2.8 2:11c82aae:::realms.fd7d09e8-a4a4-4ab2-821d-6ee3e429bf42.control:head [watch ping cookie 94076365044480] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-03-21T13:09:09.023 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:08 smithi177 podman[101991]: 2024-03-21 13:09:08.731356004 +0000 UTC m=+5.597673556 container died 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True) 2024-03-21T13:09:09.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:09 smithi177 podman[101991]: 2024-03-21 13:09:09.188430194 +0000 UTC m=+6.054747770 container cleanup 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:09:09.358 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:09 smithi177 bash[101991]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7 2024-03-21T13:09:09.789 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:09 smithi177 podman[102020]: 2024-03-21 13:09:09.456178696 +0000 UTC m=+0.716389997 container remove 5e2e973f52fa2c52d6e4801d875e9e28c93d4b136f5e1ddd01adc1a6049a8adf (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-21T13:09:10.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:09 smithi177 podman[102114]: 2024-03-21 13:09:09.790203252 +0000 UTC m=+0.093121591 container create 81229959a54ea4da37e656a34ec0eb59ae574efcf2fa8245ffab46067a3cfb5c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-21T13:09:10.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:09 smithi177 podman[102114]: 2024-03-21 13:09:09.715264358 +0000 UTC m=+0.018182697 image pull 19e12293512ff7978c8adb5f72a829060dd47f07bfd6903b6b3178d2ebc29e4e quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93 2024-03-21T13:09:10.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:10 smithi177 podman[102114]: 2024-03-21 13:09:10.026578047 +0000 UTC m=+0.329496400 container init 81229959a54ea4da37e656a34ec0eb59ae574efcf2fa8245ffab46067a3cfb5c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-21T13:09:10.108 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:10 smithi177 podman[102114]: 2024-03-21 13:09:10.030228584 +0000 UTC m=+0.333146923 container start 81229959a54ea4da37e656a34ec0eb59ae574efcf2fa8245ffab46067a3cfb5c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:09:10.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:10 smithi177 podman[102114]: 2024-03-21 13:09:10.109461106 +0000 UTC m=+0.412379467 container attach 81229959a54ea4da37e656a34ec0eb59ae574efcf2fa8245ffab46067a3cfb5c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:09:10.608 INFO:journalctl@ceph.osd.7.smithi177.stdout:Mar 21 13:09:10 smithi177 podman[102114]: 2024-03-21 13:09:10.319984439 +0000 UTC m=+0.622902779 container died 81229959a54ea4da37e656a34ec0eb59ae574efcf2fa8245ffab46067a3cfb5c (image=quay.ceph.io/ceph-ci/ceph@sha256:35f4b82cbae750007a45d4eba3ff9f653c457c283782141ef353684823d81e93, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-2024-03-21-0818-960fbbe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-21T13:09:11.018 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@osd.7.service' 2024-03-21T13:09:11.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:09:11.060 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-21T13:09:11.060 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-21T13:09:11.060 DEBUG:teuthology.orchestra.run.smithi177:> sudo systemctl stop ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a 2024-03-21T13:09:11.358 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 systemd[1]: Stopping Ceph prometheus.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.399Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-21T13:09:11.749 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.400Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.401Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.401Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a[64392]: ts=2024-03-21T13:09:11.401Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 podman[102223]: 2024-03-21 13:09:11.410309182 +0000 UTC m=+0.132629738 container died b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 podman[102223]: 2024-03-21 13:09:11.749228665 +0000 UTC m=+0.471549220 container cleanup b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:11.750 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 bash[102223]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T13:09:12.108 INFO:journalctl@ceph.prometheus.a.smithi177.stdout:Mar 21 13:09:11 smithi177 podman[102237]: 2024-03-21 13:09:11.950539398 +0000 UTC m=+0.542295733 container remove b7238dde9e6b35a9f3662386094f8d8cc5c9dca4765fbe58fdc6b8054283e85a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:12.159 DEBUG:teuthology.orchestra.run.smithi177:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-21T13:09:12.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T13:09:12.196 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-21T13:09:12.196 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T13:09:13.646 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:13.555Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T13:09:13.646 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:13.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-21T13:09:13.647 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:13 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:13.558Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi177.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.177:8443: connect: connection refused" 2024-03-21T13:09:14.146 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:13 smithi172 systemd[1]: Stopping Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:09:14.146 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:13 smithi172 podman[138788]: 2024-03-21 13:09:13.906746356 +0000 UTC m=+0.110252884 container died ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:14.461 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 podman[138788]: 2024-03-21 13:09:14.346063041 +0000 UTC m=+0.549569576 container cleanup ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:14.462 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 bash[138788]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T13:09:14.738 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 podman[138802]: 2024-03-21 13:09:14.539170509 +0000 UTC m=+0.632529983 container remove ffc0c5638d82cc0e9c768e93e74736f469751c597625b91ba69345b4af0c4d59 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:14.738 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T13:09:15.022 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:14 smithi172 systemd[1]: Stopping Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:09:15.023 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T13:09:15.023 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 systemd[1]: Stopped Ceph node-exporter.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:09:15.023 INFO:journalctl@ceph.node-exporter.a.smithi172.stdout:Mar 21 13:09:14 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 3.138s CPU time. 2024-03-21T13:09:15.282 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a[78224]: ts=2024-03-21T13:09:15.242Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T13:09:15.560 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 podman[138907]: 2024-03-21 13:09:15.283146315 +0000 UTC m=+0.208964174 container stop 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:15.560 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 podman[138907]: 2024-03-21 13:09:15.308722074 +0000 UTC m=+0.234539924 container died 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:15.860 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 podman[138907]: 2024-03-21 13:09:15.650678249 +0000 UTC m=+0.576496096 container cleanup 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:15.860 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 bash[138907]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T13:09:15.860 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 podman[138922]: 2024-03-21 13:09:15.801598517 +0000 UTC m=+0.513105588 container remove 199677338a0e5d422bb1bd5ff58619f1d3c4f0678ef95635d6e3cb2ec82ee94f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T13:09:15.860 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:15 smithi172 podman[138922]: 2024-03-21 13:09:15.860522769 +0000 UTC m=+0.572029857 volume remove d62446a4359cd26f5f8d6102d53aacb8baec97b98e4c4bc081063c865950408e 2024-03-21T13:09:16.146 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:16 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T13:09:16.146 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:16 smithi172 systemd[1]: Stopped Ceph alertmanager.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:09:16.146 INFO:journalctl@ceph.alertmanager.a.smithi172.stdout:Mar 21 13:09:16 smithi172 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.635s CPU time. 2024-03-21T13:09:39.030 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T13:09:40.794 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:40 smithi177 systemd[1]: Stopping Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:09:41.068 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:40 smithi177 podman[102633]: 2024-03-21 13:09:40.80303558 +0000 UTC m=+0.111633100 container died d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:09:41.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 podman[102633]: 2024-03-21 13:09:41.209223166 +0000 UTC m=+0.517820689 container cleanup d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:09:41.361 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 bash[102633]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T13:09:41.717 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 podman[102648]: 2024-03-21 13:09:41.418727233 +0000 UTC m=+0.619215638 container remove d7322f032b4b9bd7090538b28654c10f327b842d3f29b85cecb2c5b3b2ac8db9 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T13:09:41.717 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T13:09:41.717 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T13:09:41.717 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 systemd[1]: Stopped Ceph node-exporter.b for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:09:41.717 INFO:journalctl@ceph.node-exporter.b.smithi177.stdout:Mar 21 13:09:41 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 3.103s CPU time. 2024-03-21T13:09:42.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:41 smithi177 systemd[1]: Stopping Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715... 2024-03-21T13:09:42.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=server t=2024-03-21T13:09:41.929846643Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-21T13:09:42.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=tracing t=2024-03-21T13:09:41.93024998Z level=info msg="Closing tracing" 2024-03-21T13:09:42.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:41 smithi177 ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a[66357]: logger=ticker t=2024-03-21T13:09:41.930279379Z level=info msg=stopped last_tick=2024-03-21T13:09:40Z 2024-03-21T13:09:42.108 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:41 smithi177 podman[102757]: 2024-03-21 13:09:41.94024058 +0000 UTC m=+0.085246710 container died 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d (image=quay.io/ceph/grafana:9.4.12, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:09:42.572 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:42 smithi177 podman[102757]: 2024-03-21 13:09:42.288138608 +0000 UTC m=+0.433144736 container cleanup 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d (image=quay.io/ceph/grafana:9.4.12, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:09:42.572 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:42 smithi177 bash[102757]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a 2024-03-21T13:09:42.858 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:42 smithi177 podman[102771]: 2024-03-21 13:09:42.572583155 +0000 UTC m=+0.634402255 container remove 49853685e4f572ce496468aa9c12e0b740caaadef155276527dc664d3d61709d (image=quay.io/ceph/grafana:9.4.12, name=ceph-3bbfe794-e77e-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T13:09:42.858 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:42 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T13:09:42.858 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:42 smithi177 systemd[1]: Stopped Ceph grafana.a for 3bbfe794-e77e-11ee-95cd-87774f69a715. 2024-03-21T13:09:42.858 INFO:journalctl@ceph.grafana.a.smithi177.stdout:Mar 21 13:09:42 smithi177 systemd[1]: ceph-3bbfe794-e77e-11ee-95cd-87774f69a715@grafana.a.service: Consumed 11.551s CPU time. 2024-03-21T13:09:54.299 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:09:54.335 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:09:54.364 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-21T13:09:54.366 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/remote/smithi172/crash 2024-03-21T13:09:54.367 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/crash -- . 2024-03-21T13:09:54.418 INFO:teuthology.orchestra.run.smithi172.stderr:tar: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T13:09:54.418 INFO:teuthology.orchestra.run.smithi172.stderr:tar: Error is not recoverable: exiting now 2024-03-21T13:09:54.420 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/remote/smithi177/crash 2024-03-21T13:09:54.421 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/crash -- . 2024-03-21T13:09:54.460 INFO:teuthology.orchestra.run.smithi177.stderr:tar: /var/lib/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T13:09:54.460 INFO:teuthology.orchestra.run.smithi177.stderr:tar: Error is not recoverable: exiting now 2024-03-21T13:09:54.461 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-21T13:09:54.461 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T13:09:54.492 INFO:teuthology.orchestra.run.smithi172.stdout:2024-03-21T12:31:22.418827+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T13:09:54.494 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-21T13:09:54.495 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[SEC\]' /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T13:09:54.566 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]' /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T13:09:54.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[WRN\]' /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-21T13:09:54.707 INFO:teuthology.orchestra.run.smithi172.stdout:2024-03-21T12:31:22.418827+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-21T13:09:54.708 INFO:tasks.cephadm:Compressing logs... 2024-03-21T13:09:54.709 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T13:09:54.752 DEBUG:teuthology.orchestra.run.smithi177:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:: No such file or directory 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi172.aecdpn.log 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.a.log: /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi172.aecdpn.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.audit.log: 76.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi172.aecdpn.log.gz 2024-03-21T13:09:54.778 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-21T13:09:54.780 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-21T13:09:54.780 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-21T13:09:54.780 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-21T13:09:54.781 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T13:09:54.781 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-21T13:09:54.782 INFO:teuthology.orchestra.run.smithi177.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T13:09:54.782 INFO:teuthology.orchestra.run.smithi177.stderr:: No such file or directory 2024-03-21T13:09:54.782 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-21T13:09:54.782 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T13:09:54.782 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-21T13:09:54.784 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi177.magnlc.log 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.4.log: /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi177.magnlc.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi177.stderr: 76.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-client.rgw.foo.smithi177.magnlc.log.gz 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.5.log: --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log 2024-03-21T13:09:54.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log 2024-03-21T13:09:54.787 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.cephadm.log: 96.0% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T13:09:54.787 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 91.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T13:09:54.788 INFO:teuthology.orchestra.run.smithi177.stderr: --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T13:09:54.788 INFO:teuthology.orchestra.run.smithi177.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-21T13:09:54.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.cephadm.log: 94.6% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T13:09:54.789 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-21T13:09:54.794 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log: /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-volume.log: 89.1% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T13:09:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log: 97.1% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T13:09:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T13:09:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.2.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T13:09:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-21T13:09:54.796 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-volume.log: 93.9% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T13:09:54.804 INFO:teuthology.orchestra.run.smithi177.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T13:09:54.809 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T13:09:54.814 INFO:teuthology.orchestra.run.smithi177.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T13:09:54.939 INFO:teuthology.orchestra.run.smithi177.stderr: 89.8% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-21T13:09:55.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.1% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-21T13:09:55.371 INFO:teuthology.orchestra.run.smithi177.stderr: 92.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-21T13:09:55.557 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-21T13:09:56.219 INFO:teuthology.orchestra.run.smithi172.stderr: 93.3% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-21T13:09:56.384 INFO:teuthology.orchestra.run.smithi172.stderr: 91.0% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-21T13:09:56.433 INFO:teuthology.orchestra.run.smithi177.stderr: 93.2% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-21T13:09:56.523 INFO:teuthology.orchestra.run.smithi172.stderr: 93.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-21T13:09:56.613 INFO:teuthology.orchestra.run.smithi177.stderr: 93.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-21T13:09:56.688 INFO:teuthology.orchestra.run.smithi177.stderr: 93.6% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-21T13:09:57.019 INFO:teuthology.orchestra.run.smithi177.stderr: 93.6% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-21T13:09:57.021 INFO:teuthology.orchestra.run.smithi177.stderr: 2024-03-21T13:09:57.021 INFO:teuthology.orchestra.run.smithi177.stderr:real 0m2.254s 2024-03-21T13:09:57.021 INFO:teuthology.orchestra.run.smithi177.stderr:user 0m8.142s 2024-03-21T13:09:57.021 INFO:teuthology.orchestra.run.smithi177.stderr:sys 0m0.307s 2024-03-21T13:09:57.069 INFO:teuthology.orchestra.run.smithi172.stderr: 93.5% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-21T13:09:57.406 INFO:teuthology.orchestra.run.smithi172.stderr: 93.4% -- replaced with /var/log/ceph/3bbfe794-e77e-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-21T13:09:57.408 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-21T13:09:57.408 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m2.644s 2024-03-21T13:09:57.408 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m10.679s 2024-03-21T13:09:57.408 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.387s 2024-03-21T13:09:57.409 INFO:tasks.cephadm:Archiving logs... 2024-03-21T13:09:57.409 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/remote/smithi172/log 2024-03-21T13:09:57.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T13:09:58.968 DEBUG:teuthology.misc:Transferring archived files from smithi177:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/remote/smithi177/log 2024-03-21T13:09:58.969 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T13:10:00.276 INFO:tasks.cephadm:Removing cluster... 2024-03-21T13:10:00.276 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 --force 2024-03-21T13:10:00.571 DEBUG:teuthology.orchestra.run.smithi177:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3bbfe794-e77e-11ee-95cd-87774f69a715 --force 2024-03-21T13:10:00.863 INFO:tasks.cephadm:Removing cephadm ... 2024-03-21T13:10:00.864 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T13:10:00.879 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T13:10:00.894 INFO:tasks.cephadm:Teardown complete 2024-03-21T13:10:00.894 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T13:10:00.903 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T13:10:00.903 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:10:00.922 DEBUG:teuthology.orchestra.run.smithi177:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:10:00.933 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:10:00.939 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:10:00.939 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-21T13:10:00.940 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +1457us[+1457us] +/- 95ms 2024-03-21T13:10:00.940 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1318us[-1312us] +/- 64ms 2024-03-21T13:10:00.940 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +405us[ +405us] +/- 76ms 2024-03-21T13:10:00.940 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:10:00.947 INFO:teuthology.orchestra.run.smithi177.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:10:00.956 INFO:teuthology.orchestra.run.smithi177.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:10:00.956 INFO:teuthology.orchestra.run.smithi177.stdout:=============================================================================== 2024-03-21T13:10:00.956 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 21 +1440us[+1440us] +/- 95ms 2024-03-21T13:10:00.956 INFO:teuthology.orchestra.run.smithi177.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -1313us[-1321us] +/- 64ms 2024-03-21T13:10:00.956 INFO:teuthology.orchestra.run.smithi177.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +401us[ +401us] +/- 76ms 2024-03-21T13:10:00.956 INFO:teuthology.orchestra.run.smithi177.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:10:00.957 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T13:10:00.965 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T13:10:00.966 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T13:10:00.976 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T13:10:01.075 DEBUG:teuthology.orchestra.run.smithi177:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T13:10:01.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T13:10:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.189 DEBUG:teuthology.orchestra.run.smithi177:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-21T13:10:01.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:01.221 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T13:10:01.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T13:10:01.246 INFO:teuthology.task.internal:Duration was 2881.593850 seconds 2024-03-21T13:10:01.246 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T13:10:01.258 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T13:10:01.258 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T13:10:01.260 DEBUG:teuthology.orchestra.run.smithi177:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T13:10:01.301 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:10:01.304 INFO:teuthology.orchestra.run.smithi177.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:10:01.684 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T13:10:01.684 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-03-21T13:10:01.684 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T13:10:01.744 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi177.front.sepia.ceph.com 2024-03-21T13:10:01.744 DEBUG:teuthology.orchestra.run.smithi177:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T13:10:01.767 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T13:10:01.767 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T13:10:01.788 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T13:10:01.839 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T13:10:01.839 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T13:10:01.859 DEBUG:teuthology.orchestra.run.smithi177:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T13:10:02.511 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T13:10:02.522 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T13:10:02.523 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T13:10:02.550 DEBUG:teuthology.orchestra.run.smithi177:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T13:10:02.578 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T13:10:02.589 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:10:02.596 DEBUG:teuthology.orchestra.run.smithi177:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:10:02.619 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-03-21T13:10:02.644 INFO:teuthology.orchestra.run.smithi177.stdout:kernel.core_pattern = core 2024-03-21T13:10:02.675 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:10:02.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.706 DEBUG:teuthology.orchestra.run.smithi177:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T13:10:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:10:02.730 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T13:10:02.740 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T13:10:02.740 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/remote/smithi172 2024-03-21T13:10:02.740 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T13:10:02.807 DEBUG:teuthology.misc:Transferring archived files from smithi177:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915/remote/smithi177 2024-03-21T13:10:02.807 DEBUG:teuthology.orchestra.run.smithi177:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T13:10:02.855 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T13:10:02.855 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T13:10:02.858 DEBUG:teuthology.orchestra.run.smithi177:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T13:10:02.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T13:10:02.913 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T13:10:02.913 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T13:10:02.924 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T13:10:02.924 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T13:10:02.927 DEBUG:teuthology.orchestra.run.smithi177:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T13:10:02.939 INFO:teuthology.orchestra.run.smithi172.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 13:10 /home/ubuntu/cephtest 2024-03-21T13:10:02.954 INFO:teuthology.orchestra.run.smithi177.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 13:10 /home/ubuntu/cephtest 2024-03-21T13:10:02.955 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T13:10:02.965 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T13:10:02.986 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T13:10:03.019 DEBUG:teuthology.nuke:shortname: smithi172 2024-03-21T13:10:03.019 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:10:03.045 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:12:17.106097', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFF0nGXA1g7Xypq//zN2Ry18gLmBxWsgtx1/hZ86a45K7oECes8HVAz9MMGYVaelcKEbyfowsVmPy6gMIMmhtb0='} 2024-03-21T13:10:03.046 DEBUG:teuthology.nuke:shortname: smithi177 2024-03-21T13:10:03.046 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:10:03.073 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi177.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-03-21_12:04:17-orch:cephadm-wip-guits-testing-2024-03-21-0818-distro-default-smithi/7614915', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 12:12:17.107610', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL9zPSzfwOtgaPVEAQ34yKSvEMWmY3lOplPS0T7dkoNHJJxTZy40JHrGkyrSOUZjsbW8uwiAO8UUurct6X2Ewgs='} 2024-03-21T13:10:03.107 INFO:teuthology.orchestra.console.smithi172:Power off 2024-03-21T13:10:03.108 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T13:10:03.132 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-03-21T13:10:03.133 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:10:03.149 INFO:teuthology.orchestra.console.smithi177:Power off 2024-03-21T13:10:03.150 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T13:10:03.160 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-21T13:10:03.173 DEBUG:teuthology.orchestra.console.smithi177:power off output: Chassis Power Control: Down/Off 2024-03-21T13:10:03.173 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:10:03.199 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-21T13:10:07.161 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:10:07.201 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:10:07.272 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-21T13:10:07.312 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is on 2024-03-21T13:10:11.277 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:10:11.312 DEBUG:teuthology.orchestra.console.smithi177:pexpect command: ipmitool -H smithi177.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T13:10:11.388 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-03-21T13:10:11.423 DEBUG:teuthology.orchestra.console.smithi177:check power output: Chassis Power is off 2024-03-21T13:10:11.489 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-03-21T13:10:11.524 INFO:teuthology.orchestra.console.smithi177:Power off completed 2024-03-21T13:10:11.628 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2881.5938498973846 failure_reason: '"2024-03-21T12:31:22.418827+0000 mon.a (mon.0) 477 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log' owner: scheduled_gabrioux@teuthology success: false 2024-03-21T13:10:11.628 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T13:10:11.719 INFO:teuthology.run:FAIL